Add vLLM as a third inference engine

[cguldogan]

Description

PAIR supports two inference engines, Ollama and LM Studio. This adds vLLM as a third, on the same footing: detected or adopted, started and stopped, advertised over discovery, routed to, scheduled, and shown in the desktop and terminal interfaces.

User-visible outcome:

• vLLM appears in Engine settings on Linux with install, start, stop, restart, uninstall, and port configuration. On macOS and Windows it is listed but not installable, because vLLM publishes no GPU build there.
• An already running vLLM on the configured port (a vllm serve … or the vllm/vllm-openai container) is adopted rather than restarted. That is the normal path.
• A vLLM process serves one model chosen before it starts, so Engine settings gains a Model to serve field (a Hugging Face id). Starting vLLM with no model configured fails with a message pointing at the setting instead of a launch error. There are no pull, load, eject, or delete operations.
• vLLM shares the existing OpenAI-compatible proxy with LM Studio: one listener, one port, both engines' models in a single /v1/models. A peer's request is routed to whichever local engine owns the requested model.
• Manually added nodes are probed for vLLM on port 8000 (distinguished from other OpenAI-compatible servers by GET /version). The scheduler ranks vLLM nodes with the same load signal as the other engines.

Scope

Included: engine manifest (services/nvpair-engine-manager/manifests/vllm.json, Linux amd64/arm64), the runtime.model / runtime.extra_args manifest fields and the persistent engine:set-model method, discovery key vl, the OpenAI-compatible proxy generalised to any OpenAI-compatible engine (binary name and relay namespace unchanged; README retitled), broker advertise loop and manual-node bridge, scheduler engine list, TUI labels, desktop engine type and capabilities, docs.

Excluded: a managed port facade for vLLM (it keeps port 8000 and is never moved), model download or deletion for vLLM, and any Windows or macOS install path.

Breaking change for API consumers of lmstudio-proxy: node/add-manual and node/remove-manual now require an engine field (lmstudio or vllm), hence its MAJOR bump.

Related: a sibling PR makes nodes added by address first-class peers on overlay networks. Both touch services/nvpair-manual-nodes/manager.go and services/versions.json; whichever merges second needs a small conflict resolution (vLLM becomes a third engine leg of the bare-host probe, and the higher version of each component is kept).

Validation

Environment: macOS 15 arm64, Go 1.27, Node 23.

cd services && ./build.sh                       # 13 binaries, product 0.92.0
go test ./...   # nvpair-engine-manager, lmstudio-proxy, nvpair-ui-broker, shared, nvpair-manual-nodes, nvpair-job-scheduler, nvpair-tui
cd services/tests && go test ./...              # pass, 0 skips
cd desktop && npm run typecheck && npm run lint && npm run test:unit && npm run dead-code:check && npm run service-contracts:write && npm run service-contracts:check
node scripts/spdx-headers.mjs                   # 0 missing

All pass, except nvpair-engine-manager TestUninstallTerminatesRunningInstance, which fails identically on a pristine main on this host (it cannot resolve the image path of the process holding a port) and is unrelated.

End-to-end: a stub serving vLLM's /health, /version, /v1/models, and /v1/chat/completions was adopted by the real broker (engine:get-installed reported it running on its port with the configured model), the OpenAI proxy listed its model and completed a chat request, and the workload was tagged vllm. Adoption of a real two-node vLLM (--tensor-parallel-size 2 --nnodes 2) on Linux arm64 has also been exercised. The manifest's install block and the 30-minute readiness budget have not been run on real Linux GPU hardware.

Risk

• Compatibility: the engine field on the proxy's manual-node methods is required (MAJOR bump on lmstudio-proxy). EngineStatus gains an optional model. Everything else is additive.
• Security: unchanged model. The managed manifest starts vLLM with --host 127.0.0.1; an adopted instance keeps whatever bind it was started with, exactly as an adopted Ollama does.
• Packaging: no new binary. Manifests are embedded.

Versions: lmstudio-proxy 0.16.2→1.0.0, nvpair-engine-manager 0.17.4→0.18.0, nvpair-ui-broker 0.40.2→0.41.0, nvpair-manual-nodes 0.11.1→0.12.0, nvpair-job-scheduler 0.4.1→0.5.0, patch bumps for nvpair-node-scanner, nvpair-errors, nvpair-workload-manager, nvpair-tui; product 0.91.7→0.92.0.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

Related: #10 (touches the same manual-nodes file; see the note under Scope) and #11.

[vedcsolution]

+1 — this would be a big deal for DGX Spark owners specifically.

Our setup: a small home/lab cluster of NVIDIA GB10 (DGX Spark) nodes paired in PAIR, each serving production models via vllm serve on :8000 (OpenAI-compatible, e.g. NVFP4-quantized Qwen and GLM MoE models with long context). Ollama and LM Studio don't really cover these: we need the vLLM-specific runtime options (quantization formats like NVFP4, speculative decoding, PLE/offload patches, multi-node TP) that aren't exposed through the other engines.

The mismatch today is that PAIR sees and manages the nodes (pairing, health, workload signals all work great) but the engine-manager only probes the fixed Ollama/LM Studio ports, so a healthy vllm serve is completely invisible — the Overview shows all nodes with no downloadable engines and no way to route to the models that actually live on them. We currently keep a separate LiteLLM gateway in front of the vLLM endpoints, which works but means PAIR can't do what it's uniquely good at: cluster-aware routing with node GPU/memory telemetry.

The issue body's "adopt an already-running vLLM on the configured port" framing is exactly what we'd use — we don't need PAIR to install or lifecycle-manage vLLM (systemd/docker handles that), just detect /v1/models on a configurable port per node, advertise the models over discovery, and route to them through the OpenAI-compatible endpoint. That "adopt, don't install" subset alone would unblock mixed clusters like ours.

Happy to test a preview build on a 6-node all-GB10 cluster if that's useful.

[cguldogan]

+1 — this would be a big deal for DGX Spark owners specifically.

Our setup: a small home/lab cluster of NVIDIA GB10 (DGX Spark) nodes paired in PAIR, each serving production models via vllm serve on :8000 (OpenAI-compatible, e.g. NVFP4-quantized Qwen and GLM MoE models with long context). Ollama and LM Studio don't really cover these: we need the vLLM-specific runtime options (quantization formats like NVFP4, speculative decoding, PLE/offload patches, multi-node TP) that aren't exposed through the other engines.

The mismatch today is that PAIR sees and manages the nodes (pairing, health, workload signals all work great) but the engine-manager only probes the fixed Ollama/LM Studio ports, so a healthy vllm serve is completely invisible — the Overview shows all nodes with no downloadable engines and no way to route to the models that actually live on them. We currently keep a separate LiteLLM gateway in front of the vLLM endpoints, which works but means PAIR can't do what it's uniquely good at: cluster-aware routing with node GPU/memory telemetry.

The issue body's "adopt an already-running vLLM on the configured port" framing is exactly what we'd use — we don't need PAIR to install or lifecycle-manage vLLM (systemd/docker handles that), just detect /v1/models on a configurable port per node, advertise the models over discovery, and route to them through the OpenAI-compatible endpoint. That "adopt, don't install" subset alone would unblock mixed clusters like ours.

Happy to test a preview build on a 6-node all-GB10 cluster if that's useful.

full implementation is here I would love to see it :)

https://github.com/cguldogan/Personal-AI-Router/releases

[arcaven]

I built and ran the suite at a4b96a3 on macOS arm64, since this adds an engine and a second platform data point seemed worth having. The vLLM work passes; the two failures I hit are pre-existing on main and are not from this PR.

Environment: darwin/arm64 (Apple Silicon), Go 1.26.5, Node 25.9.0, clean tree at a4b96a3.

Desktop (make test-desktop): 39 files, 220 tests, all pass, including the new vllm-engine.test.ts (6) and openai-proxy-engines.test.ts (6).

Go: make test-services halts at the first failing module, so I ran each module separately. 18 modules, 16 pass, 2 fail:

module	test	also fails at origin/main 13b6811
nvpair-engine-manager	TestUninstallTerminatesRunningInstance	yes
ollama-proxy	TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone	yes

Both reproduce at main with identical messages, so neither is caused by this change. lmstudio-proxy, which carries most of the new engine code plus multiengine_test.go, passes.

On those two, in case it saves anyone time:

• The alias test binds 127.0.0.2. macOS configures only 127.0.0.1 on lo0, where Linux treats all of 127.0.0.0/8 as local, so the bind fails before the test's own logic runs. An ifconfig lo0 alias in the harness or a platform skip would cover it.
• TestUninstallTerminatesRunningInstance may reach a real macOS gap rather than a test-only one. procImage in proc_unix.go reads /proc/<pid>/exe, which macOS does not provide, so it returns ""; isOurEngineImage returns false for an empty image; and the orphan-reclaim branch in doStop is skipped as a result. That would leave the path described in its own comment, letting a user's OFF take effect "instead of being refused forever", unreachable on macOS. That is my reading of the source rather than something I instrumented. Happy to file it separately if useful.

Two small notes on the manifest, neither of them asks:

• manifests/vllm.json declares linux/amd64 and linux/arm64 only, which I read as deliberate. It does mean an Apple Silicon tester can exercise the Go and TS layers but not the runtime path.
• The manifest exposes /v1/models and /v1/chat/completions. vLLM also serves the Anthropic Messages API (vllm/entrypoints/anthropic/api_router.py), which is the subject of [Feature]: Expose the Anthropic Messages API (POST /v1/messages) on the PAIR proxy #16. Nothing this PR needs to take on; noting it only because the two touch.

[arcaven]

Filed the macOS piece separately as #17, since it reproduces at origin/main and is a different root cause from anything here. Nothing in it blocks this PR.

[arcaven]

Small one from the same pass: three files in this PR are not gofmt-clean.

$ git checkout a4b96a3 && gofmt -l services/ scripts/
services/nvpair-engine-manager/registry.go
services/nvpair-node-info/stats_windows.go
services/shared/noderec/noderec.go
services/shared/splitlisten/splitlisten_test.go
services/tests/model_routing_interop_test.go

$ git checkout 13b6811 && gofmt -l services/ scripts/     # merge base
services/nvpair-node-info/stats_windows.go
services/shared/splitlisten/splitlisten_test.go

So registry.go, noderec.go, and model_routing_interop_test.go are the three this branch introduces; the other two are already on main and are not yours. In registry.go it is struct tag alignment in Runtime, where the new ExtraArgs field widens the column and the fields after it were not realigned. gofmt -w on the three fixes it.

Two other notes from testing this branch on Apple Silicon, so they are not mistaken for anything here:

• The lmstudio-proxy failure I mentioned earlier turned out to be machine state, not this PR and not main. It is [Bug]: tests write the live per-user app-data directory, so a second suite run fails #48.
• I opened Skip the orphan-reclaim test where the listener image is unresolvable #47 for the nvpair-engine-manager test that fails on macOS at main, which is the one from my first comment.

[jlacroix82]

I've built on this PR at a4b96a3 to add SGLang alongside vLLM for our existing cluster deployments. The work is in my fork, keeping this PR's vLLM implementation and shared OpenAI proxy intact:

https://github.com/jlacroix82/Personal-AI-Router/tree/feat/vllm-sglang

Extension commit: jlacroix82@58508c6

The extension adds:

• SGLang desktop engine registration, node visibility, model inventory, and proxy attribution.
• An operator-installed SGLang manifest, default port 30000, and adoption of an already-running process. No new installer or runtime dependency is introduced.
• The sg discovery key through the existing OpenAI proxy, separate local engine state, and scheduler attribution.
• Bare-host SGLang detection using /get_model_info plus /v1/models, with sglang_* status fields relayed by the broker.
• Local port/model configuration before detection, so an existing server on a non-default port can be connected from Engine settings.

I also fixed the broker's manualToEnriched combined model list to include vLLM and SGLang; previously it combined only Ollama and LM Studio, although the per-engine map included vLLM.

Validation on Linux ARM64, Node 26.5.0, Go 1.26.5:

• Desktop typecheck: pass; unit tests: 40 files / 222 tests pass.
• Lint: no errors, one pre-existing formatting warning in node-info-poller.ts; dead-code and service-contract checks pass.
• go test ./... passes in shared, engine-manager, OpenAI proxy, UI broker, manual-nodes, and job-scheduler. Added tests cover SGLang adoption, identity/inventory probing, proxy advertisement and withdrawal, and HTTP ingress to separate vLLM/SGLang servers.
• All 13 service binaries and the Linux ARM64 Electron preview build/package successfully.
• Extended the existing cross-process strict-routing test with vLLM and SGLang cases; execution skipped because the host's PAIR default ports were occupied. This is not counted as a pass for that scenario.

This is stub-backed validation, not a claim of a completed production-cluster rollout. Native single-model controls are retained; externally managed processes remain under their supervisor. Authenticated engine endpoints and fully dynamic arbitrary-engine registration are still outside this change.

The adapter repository now also has a separate SGLang manifest/configuration, non-conflicting wrapper/child ports, two passing mock-adapter tests, and GUI setup instructions:
https://github.com/jlacroix82/pair-multi-engine

Happy for the SGLang follow-up to stay separate from #9. The branch is available to review or cherry-pick once the vLLM foundation is settled.

[NV-sschneider]

Thanks all for your contributions to the project! We are just getting back from the long weekend and will be reviewing this PR and all the others as well.

We are very excited about vLLM support!

Hang tight. :)

[cguldogan]

I've built on this PR at a4b96a3 to add SGLang alongside vLLM for our existing cluster deployments. The work is in my fork, keeping this PR's vLLM implementation and shared OpenAI proxy intact:

https://github.com/jlacroix82/Personal-AI-Router/tree/feat/vllm-sglang

Extension commit: jlacroix82@58508c6

The extension adds:

• SGLang desktop engine registration, node visibility, model inventory, and proxy attribution.
• An operator-installed SGLang manifest, default port 30000, and adoption of an already-running process. No new installer or runtime dependency is introduced.
• The sg discovery key through the existing OpenAI proxy, separate local engine state, and scheduler attribution.
• Bare-host SGLang detection using /get_model_info plus /v1/models, with sglang_* status fields relayed by the broker.
• Local port/model configuration before detection, so an existing server on a non-default port can be connected from Engine settings.

I also fixed the broker's manualToEnriched combined model list to include vLLM and SGLang; previously it combined only Ollama and LM Studio, although the per-engine map included vLLM.

Validation on Linux ARM64, Node 26.5.0, Go 1.26.5:

• Desktop typecheck: pass; unit tests: 40 files / 222 tests pass.
• Lint: no errors, one pre-existing formatting warning in node-info-poller.ts; dead-code and service-contract checks pass.
• go test ./... passes in shared, engine-manager, OpenAI proxy, UI broker, manual-nodes, and job-scheduler. Added tests cover SGLang adoption, identity/inventory probing, proxy advertisement and withdrawal, and HTTP ingress to separate vLLM/SGLang servers.
• All 13 service binaries and the Linux ARM64 Electron preview build/package successfully.
• Extended the existing cross-process strict-routing test with vLLM and SGLang cases; execution skipped because the host's PAIR default ports were occupied. This is not counted as a pass for that scenario.

This is stub-backed validation, not a claim of a completed production-cluster rollout. Native single-model controls are retained; externally managed processes remain under their supervisor. Authenticated engine endpoints and fully dynamic arbitrary-engine registration are still outside this change.

The adapter repository now also has a separate SGLang manifest/configuration, non-conflicting wrapper/child ports, two passing mock-adapter tests, and GUI setup instructions: https://github.com/jlacroix82/pair-multi-engine

Happy for the SGLang follow-up to stay separate from #9. The branch is available to review or cherry-pick once the vLLM foundation is settled.

I saw this after the SGLang PR had already been submitted. You could open a PR as well maybe your version has a better chance of being accepted?

[jlacroix82]

Follow-up on the vLLM/SGLang GUI branch: testing against an existing native deployment turned up a few things that needed tightening up.

Changes are in 21a363d on feat/vllm-sglang:

• External processes are labeled as externally managed. Desktop and TUI no longer offer lifecycle controls for them; declared deployments also reject lifecycle/model/port mutations in the Go service.
• Native adoption checks engine identity as well as health (/version for vLLM, /get_model_info for SGLang). The broker no longer guesses a native endpoint when engine-manager is unavailable.
• An operator-maintained engine-deployments.json describes API heads, workers and capabilities. Workers aren't separate routes for that engine, and their GPU pressure contributes to the head's scheduling load. This is explicit configuration, not automatic topology discovery or PAIR-managed tensor parallelism.
• Embedding-only deployments retain their model inventory but reject chat/completion routing. Capability metadata reaches inference-dispatcher and the generation demo.
• The GUI model parser now accepts OpenAI data[].id inventories. It previously understood Ollama and native LM Studio responses but missed the native vLLM/SGLang shape.
• Engine settings have a connection/status refresh control, and model catalogs distinguish available models from confirmed loaded models.
• Fixed a shared-proxy scheduling issue: vLLM, SGLang and LM Studio must not race to overwrite the same proxy priority store.

Validation: 224 desktop unit tests pass; typecheck, contracts and dead-code checks pass. Lint has no errors (one existing formatting warning). Go suites pass for shared, engine-manager, broker, OpenAI proxy, scheduler and TUI; inference-dispatcher tests and both adapter tests pass. ARM64 packaging succeeds. A live native vLLM embedding endpoint is detected as external/loaded, retains embedding metadata through PAIR, and rejects a chat request at the proxy. The model container was not restarted.

SGLang adoption/routing has simulated-server coverage, not a new live SGLang GPU benchmark. Remote fleet rollout and controlled throughput benchmarking are still separate steps. Participating nodes need matching preview services and deployment policy; stock PAIR does not acquire this behavior from a manifest alone.

Setup and limits. The optional adapter repo also has updated identity overrides and setup notes in a40049a.

Service versions: engine-manager 0.20.0, OpenAI proxy 1.2.0, scheduler 0.7.0, broker 0.42.1 and TUI 0.8.1. These changes are on my fork branch; this comment links the follow-up rather than modifying the PR author's source branch.

[jlacroix82]

Another follow-up from testing this against the running Spark cluster: the UI was mixing up “PAIR owns this process” with “this process is running.” It also treated distributed workers with no HTTP server as stopped.

The fix is on my fork: ff26f5b, branch feat/vllm-sglang.

• External engines keep a checked, read-only switch when running, instead of losing the switch entirely.
• Node rows identify the deployment and head/worker role. Settings separate runtime activity from API readiness.
• An optional containers map in engine-deployments.json gives each member an exact local Docker container to inspect. The check is bounded and read-only; it doesn't depend on which vLLM/SGLang image or launcher CLI is used.
• Workers can be running with no local API. They remain excluded from inference routing. Missing/unobservable worker state and failed peer status polls show Unknown, not Off.

I rolled the matching PAIR services and policy onto eight cluster devices hosting three distributed vLLM deployments, plus the local embedding host. The live GUI reports all nine vLLM switches checked and read-only, with three cluster API heads and five workers. Model-container IDs, start times and PIDs stayed unchanged. No model containers were restarted and no inference benchmark load was sent.

Validation: 226 desktop unit tests; shared, engine-manager, broker, scheduler, OpenAI proxy and TUI Go suites; typecheck, contracts, dead-code and SPDX checks. Lint has no errors and one existing unrelated formatting warning. Two ingress tests now isolate their config so a developer's actual deployment policy can't change their expected results.

One limit worth being explicit about: Docker “running” describes the container, not the health of every child process. Head API readiness is still checked separately. Non-Docker workers remain unknown until a suitable observer exists. SGLang follows the same code path and has test coverage, but I don't currently have a live SGLang deployment for a GPU integration run.

Setup and wire-field details: external deployments. This is a follow-up commit for the PR author to pick up; it doesn't change the PR's source branch directly.