[Feature]: Adopt arbitrary OpenAI-compatible backends with endpoint manifests, model aliases, and deterministic capability routing

[darkmatter2222]

Area

Desktop experience

User problem

Area

Engine or model management

User problem

PAIR solves much of the difficult infrastructure required for distributed local inference: node discovery, trust, model inventory, routing, workload visibility, and failover.

The remaining limitation for heterogeneous performance-oriented clusters is that inference backends still need to be explicitly supported as PAIR engines.

In practice, different hardware often performs best with different inference runtimes and configurations. Requiring an operator to replace an already optimized backend with a PAIR-supported engine can negate much of the reason for building a heterogeneous cluster.

I have a working three-node example:

• RTX 5090 — vLLM, Qwen3.8-27B NVFP4, primary/high-throughput node
• RTX 3090 — llama.cpp, Qwen3.8-27B Q4_K_M GGUF, secondary node
• DGX Spark / GB10 — large MoE / VLM endpoint, overflow and vision-specialist node

The router currently used for this cluster is open source here:

https://github.com/darkmatter2222/HomeAILab/tree/main/router

Its declarative endpoint configuration is here:

https://github.com/darkmatter2222/HomeAILab/blob/main/router/config/router.yaml

And the request classification / deterministic selection implementation is here:

https://github.com/darkmatter2222/HomeAILab/blob/main/router/internal/routing/classify.go

This is not intended as a competing implementation or a request for PAIR to copy this router. It is a working reference showing the behaviors that currently prevent this deployment from moving entirely to PAIR.

The core issue is broader than adding one more runtime.

PR #9 adds vLLM as another first-class engine, which is useful. But vLLM, SGLang, llama.cpp server, TensorRT-LLM gateways, custom inference servers, and future engines increasingly expose substantially the same OpenAI-compatible inference surface.

Ideally, every compatible runtime should not require a bespoke PAIR engine implementation before it can participate in the cluster.

There is also a heterogeneous routing problem.

Two nodes may expose the same logical model while having radically different:

• hardware
• quantization
• inference runtime
• context capacity
• concurrency
• prefill performance
• generation performance
• modality support
• cold-start / TTFT behavior

A request should therefore be matched against endpoint capabilities before ordinary load-based scheduling occurs.

Real-world reference behavior

The router above currently models endpoints declaratively.

For example, the same logical coding workload can be served by:

RTX 5090
vLLM
Qwen3.8-27B NVFP4
262K context
priority 10
static capacity 1

    ↓ overflow

RTX 3090
llama.cpp
Qwen3.8-27B Q4_K_M
262K context
priority 20
static capacity 1

    ↓ overflow

DGX Spark
large Flash-Next / VLM backend
262K context
priority 30
static capacity 3
vision capable
long-prefill behavior

The router does not query GPU utilization, Prometheus, Docker, model inventory, or health synchronously for every inference request.

Those signals are maintained asynchronously.

The inference hot path approximately does:

classify request
↓
determine required capabilities
↓
determine required context
↓
read in-memory endpoint table
↓
reject incompatible endpoints
↓
sort eligible endpoints deterministically
↓
atomically reserve endpoint capacity
↓
stream directly to backend

Request classification currently includes:

• image/vision requirement
• tool requirement
• streaming requirement
• estimated context requirement
• requested maximum output
• logical model alias

Candidate exclusion includes:

• vision unsupported
• tools unsupported
• streaming unsupported
• insufficient context
• unhealthy
• disabled
• draining
• capacity full

Only after eligibility is established is endpoint preference evaluated.

This distinction is important in heterogeneous clusters: eligibility first, preference/load second.

Desired outcome

PAIR should support an externally managed / bring-your-own OpenAI-compatible engine.

An operator would be able to register an already-running inference endpoint without PAIR needing to understand or control that engine's process lifecycle.

At minimum, an external endpoint could declare:

endpoint:
api_family: openai
base_url: http://127.0.0.1:8201

model:
physical_name: qwen3.8
logical_alias: local-coding

capabilities:
text: true
vision: false
tools: true
streaming: true

context:
max_tokens: 262144

routing:
priority: 10
capacity: 1

health:
path: /health

PAIR would handle discovery, trust, routing, streaming, failover, and cluster visibility, while the external runtime remains under operator control.

1. Generic externally managed engines

Support registering an arbitrary OpenAI-compatible server.

Examples include:

• vLLM
• SGLang
• llama.cpp server
• TensorRT-LLM gateways
• other OpenAI-compatible runtimes

PAIR would not need to:

• install it
• launch it
• terminate it
• load its model
• unload its model
• change its runtime configuration

It would simply adopt the endpoint as inference capacity.

First-class managed engines such as Ollama, LM Studio, and vLLM could continue to provide the richer lifecycle experience.

2. Logical model aliases

Allow multiple physically different deployments to advertise one logical model.

For example:

PAIR model: local-coding

5090:
upstream model = qwen3.8

3090:
upstream model = qwen3.8

Spark:
upstream model = another physical/model deployment

The client should not need to know runtime-specific model IDs or quantization names.

This is particularly useful when the same logical model is represented differently across Hugging Face, GGUF, TensorRT, or runtime-specific deployments.

3. Declarative endpoint capabilities

Allow an endpoint/model combination to advertise optional characteristics such as:

• text
• vision
• tools
• streaming
• reasoning/thinking
• maximum context
• quantization
• runtime type
• GPU class
• configured concurrency/capacity
• optional workload tags

These should act as eligibility constraints.

For example, an image request should never be routed to an endpoint declaring:

vision: false

even if that node currently has the lowest GPU utilization.

Similarly, a request requiring 180K context should not be dispatched to an endpoint characterized for only 128K.

4. Static capacity / admission control

Allow an operator to state how many simultaneous requests a particular deployment should accept.

For example:

5090 / vLLM capacity = 4
3090 / llama.cpp capacity = 1
Spark capacity = 3

The runtime may technically allow more requests, but the operator may have benchmarked a lower concurrency as the point that provides acceptable TTFT or throughput.

PAIR should be able to stop assigning work to that endpoint when its configured admission capacity is reached.

This is complementary to GPU-utilization/load signals rather than a replacement for them.

5. Deterministic baseline preference

Allow an optional per-endpoint or per-model priority.

For example:

RTX 5090 priority 10
RTX 3090 priority 20
DGX Spark priority 30

Meaning:

use 5090 while eligible and capacity is available
↓
then 3090
↓
then Spark

Dynamic scheduling signals can still operate within or on top of this policy.

This is important when heterogeneous systems have very different performance characteristics.

An idle slower endpoint should not necessarily receive a request merely because it currently has lower GPU utilization than a substantially faster endpoint.

This is related to #7, which requests richer heterogeneous placement signals. The request here is specifically for the endpoint contract and declarative information necessary to express such a deployment.

6. Request capability classification

Ideally PAIR would classify enough of the request to establish eligibility before scheduling.

At minimum:

requested model
vision/images?
tools?
streaming?
required context?

More advanced scheduling signals could be added independently.

A simple eligibility pipeline could be:

request
↓
model ownership / alias
↓
API compatibility
↓
modality
↓
tool support
↓
context requirement
↓
health
↓
configured capacity
↓
scheduler / preference

7. Per-endpoint timeout characteristics

Different inference platforms can have dramatically different prefill behavior.

For example, the DGX Spark endpoint in the reference deployment can experience a very long cold TTFT while loading/prefilling a large MoE model.

The 5090 and 3090 should fail quickly when genuinely unavailable, while the Spark may legitimately require a much longer first-byte/response-header timeout.

Ideally this could be declared per endpoint:

timeouts:
connect_ms: 250
first_byte_ms: 180000

rather than increasing the timeout globally for every backend.

8. Keep the inference hot path lightweight

One aspect of the reference implementation that has worked particularly well is separating the control plane from the inference data plane.

Health checks, discovery, telemetry, model inspection, and GPU statistics happen asynchronously.

The request path reads previously validated state from memory, reserves capacity, and immediately connects to the selected upstream.

A similar architectural property in PAIR would be valuable as clusters and agent concurrency increase.

Why this complements existing PAIR work

This request intentionally does not duplicate several existing efforts.

#7 — External / heterogeneous scheduling hooks and richer placement signals

#7

#7 addresses how placement can become more intelligent.

This request addresses the declarative endpoint/capability layer that makes a heterogeneous externally managed runtime eligible for that scheduler in the first place.

PR #9 — Add vLLM as a third inference engine

#9

PR #9 is valuable and demonstrates that additional runtimes are useful.

The proposed generic external-engine contract would complement first-class vLLM support rather than replace it:

Managed vLLM
→ PAIR owns lifecycle and provides full integration

External vLLM
→ advanced operator owns lifecycle
→ PAIR only adopts/routs it

External SGLang
External llama.cpp
External future runtime
→ same generic contract

This would prevent PAIR from needing a bespoke lifecycle implementation for every inference server before that server can participate in routing.

Anthropic "/v1/messages" support is also already being discussed separately in #16, so this request intentionally does not make that protocol the focus.

Alternatives considered

Replace the optimized runtimes with Ollama or LM Studio

This simplifies cluster management, but different hardware may achieve materially better performance with different inference runtimes, model formats, speculative decoding strategies, KV-cache configurations, and quantizations.

Add every inference runtime individually to PAIR

This provides excellent first-class integrations but creates ongoing maintenance requirements for every new engine.

A generic OpenAI-compatible adoption contract would provide a common denominator while preserving richer integrations where available.

Continue running an independent router in front of PAIR

The current reference router already implements the required heterogeneous placement behavior.

However, doing so duplicates functionality PAIR already provides or is better positioned to provide:

• discovery
• trust/mTLS
• cluster membership
• workload visibility
• node availability
• failover
• model inventory

The ideal outcome would be to use PAIR as the cluster control plane and retire most of the custom router.

Route directly to individual inference servers

This loses the primary benefits of PAIR and forces clients to understand cluster topology.

Compatibility and security implications

The feature could be entirely opt-in.

Existing Ollama, LM Studio, vLLM, and other managed-engine behavior would remain unchanged.

Externally managed endpoints should preferably remain bound to loopback on their host, with PAIR retaining responsibility for trusted inter-node transport.

PAIR should not distribute local engine credentials to peers.

Capability metadata contains operational characteristics only and should contain no prompt or response content.

An external endpoint that becomes unhealthy should simply become ineligible for new routing.

Because it is explicitly marked externally managed, PAIR should never stop, restart, reconfigure, unload, or otherwise mutate that process unless the operator opts into such management.

Validation approach

A useful integration test would use three heterogeneous nodes.

Node A

Runtime: vLLM
Hardware: RTX 5090
Logical model: local-coding
Physical model: qwen3.8
Context: 262144
Vision: false
Tools: true
Capacity: 2
Priority: 10

Node B

Runtime: llama.cpp server
Hardware: RTX 3090
Logical model: local-coding
Physical model: qwen3.8
Context: 262144
Vision: false
Tools: true
Capacity: 1
Priority: 20

Node C

Runtime: arbitrary OpenAI-compatible VLM
Hardware: DGX Spark
Logical model: local-coding
Context: 262144
Vision: true
Tools: true
Capacity: 3
Priority: 30
Long first-token timeout

Validate that:

1. All three can be registered without PAIR controlling their process lifecycle.
2. One logical model is exposed to clients.
3. Normal text requests prefer Node A.
4. When Node A reaches configured capacity, routing spills to Node B.
5. When A and B are unavailable/full, text can spill to C if allowed.
6. Image requests exclude A and B before scheduling and route to C.
7. Requests exceeding a node's declared context exclude that node before dispatch.
8. Stopping one upstream removes it from eligibility without affecting healthy peers.
9. Streaming remains streamed end-to-end.
10. Long first-token behavior on C does not require increasing timeouts for A and B.
11. No engine-local credential crosses the PAIR trust boundary.
12. Existing clients and existing PAIR-managed engines behave exactly as before.

Reference implementation

A working implementation of most of the requested routing semantics is available here:

Router

https://github.com/darkmatter2222/HomeAILab/tree/main/router

Declarative heterogeneous endpoint configuration

https://github.com/darkmatter2222/HomeAILab/blob/main/router/config/router.yaml

Capability-aware deterministic request selection

https://github.com/darkmatter2222/HomeAILab/blob/main/router/internal/routing/classify.go

The project is not presented as the desired PAIR implementation. It is provided as a concrete example of the workload and routing requirements and as testable reference behavior.

If PAIR supported this external endpoint/capability contract, this custom routing layer could largely be replaced by PAIR while retaining the optimized inference runtime on each GPU.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

Desired outcome

PAIR should support a generic externally managed OpenAI-compatible endpoint that can participate in cluster routing without requiring PAIR to own or manage that runtime’s lifecycle. An operator should be able to register an existing vLLM, SGLang, llama.cpp, TensorRT-LLM, or other compatible inference server and declare the information PAIR needs for safe routing, including logical model alias, upstream model name, text/vision capability, tool and streaming support, maximum context, configured concurrency/capacity, routing priority, health endpoint, and per-endpoint timeouts.

PAIR should use those declarations as eligibility constraints before scheduling. A vision request should never be sent to a text-only endpoint, a request requiring more context than an endpoint supports should exclude that endpoint, and an endpoint at its configured capacity should not receive additional work. For heterogeneous clusters, operators should also be able to define a deterministic baseline preference such as RTX 5090 → RTX 3090 → DGX Spark while still allowing PAIR to fail over when a preferred endpoint is unavailable or full.

Multiple physical deployments should also be able to advertise the same logical model even when they use different runtimes, quantizations, or upstream model identifiers, so clients only need to request a stable alias such as "local-coding".

The desired result is for PAIR to remain responsible for cluster-level capabilities such as discovery, trust/mTLS, routing, failover, model inventory, telemetry, and workload visibility, while advanced users retain full control over the inference runtime and optimization strategy on each machine. In a heterogeneous deployment like the referenced RTX 5090 + RTX 3090 + DGX Spark cluster, this would allow PAIR to replace most of the custom routing layer without sacrificing the tuned vLLM, llama.cpp, or specialized VLM backends already optimized for each device.

Alternatives considered

The primary alternative is to continue using the custom HomeAILab router I already built for this exact heterogeneous inference use case: https://github.com/darkmatter2222/HomeAILab/tree/main/router. That router already sits in front of my RTX 5090, RTX 3090, and DGX Spark and provides the behaviors I need today, including deterministic priority routing, static per-endpoint capacity, capability-aware eligibility, context-window filtering, vision/tool/streaming awareness, logical model aliasing, asynchronous health checks, direct streaming, endpoint draining, and failover across different inference runtimes. The declarative configuration is here: https://github.com/darkmatter2222/HomeAILab/blob/main/router/config/router.yaml, and the request classification and deterministic endpoint selection logic is here: https://github.com/darkmatter2222/HomeAILab/blob/main/router/internal/routing/classify.go. Continuing with that router is viable and gives me complete control over how different hardware and runtimes are used—for example, preferring a tuned vLLM deployment on the RTX 5090, then a llama.cpp deployment on the RTX 3090, and finally using the DGX Spark as overflow and for specialized VLM workloads. The downside is that maintaining a separate router means continuing to own functionality that PAIR already provides or is better positioned to provide, including node discovery, secure peer trust/mTLS, cluster membership, model inventory aggregation, workload visibility, node availability, failover, and the broader user experience around managing a distributed personal AI cluster. In other words, the custom router proves that these routing semantics are useful and practical, but it also highlights the duplication. The preferred direction would be to let PAIR absorb the cluster-control responsibilities while preserving the ability to attach tuned, externally managed inference runtimes with declarative capabilities, capacity, priority, and timeout characteristics. If PAIR could do that, most of the custom HomeAILab routing layer could be retired rather than maintained in parallel, while still preserving the performance and specialization benefits that motivated the custom router in the first place.

Compatibility and security implications

This should be implemented as an additive, opt-in capability so existing PAIR behavior for Ollama, LM Studio, vLLM, and other managed engines remains unchanged. Externally managed endpoints should participate in routing only when explicitly configured, and PAIR should treat them as inference capacity rather than assuming control over their process lifecycle. Ideally, those runtimes remain bound to loopback or another restricted local interface, with PAIR continuing to provide the trusted inter-node transport and mTLS boundary. Any credentials required by an external engine should remain local to the node hosting that engine and must not be propagated to peers, exposed through discovery, or written to logs. Declarative endpoint metadata such as model aliases, context limits, modality support, capacity, priority, health paths, and timeout characteristics contains operational information only and should never include prompt or response content. Logical model aliases should map only to explicitly configured upstream model identifiers so clients cannot use them to select arbitrary hosts or bypass normal PAIR eligibility and trust checks. If an externally managed endpoint becomes unhealthy or unavailable, it should simply become ineligible for new requests and normal PAIR failover behavior should apply. Because these endpoints are explicitly marked as externally managed, PAIR should not stop, restart, unload, reconfigure, or otherwise mutate the underlying inference server unless the operator separately opts into lifecycle management. Existing clients should continue to use the same PAIR APIs without modification, and deployments that do not configure external endpoints should see no behavioral or security change.

Validation approach

A practical validation approach would use a small heterogeneous three-node cluster that mirrors the reference HomeAILab deployment: an RTX 5090 running vLLM, an RTX 3090 running llama.cpp, and a DGX Spark running a specialized OpenAI-compatible VLM backend. Each endpoint should be registered as externally managed and given explicit metadata for logical model alias, physical model name, context limit, modality support, tool support, streaming support, static capacity, priority, health path, and timeout characteristics. Validation should first confirm that PAIR discovers and routes to all three endpoints without attempting to install, start, stop, unload, or otherwise manage their processes. A single logical model alias should then be exposed to clients even though the underlying runtimes and physical model identifiers differ. Normal text requests should prefer the highest-priority eligible endpoint, then spill to lower-priority nodes only when the preferred endpoint is unavailable or at its configured capacity. Vision requests should exclude text-only endpoints before scheduling, context requirements larger than a node's declared limit should exclude that node, and endpoints marked unhealthy, disabled, draining, or full should not receive new work. Streaming and tool-calling requests should be verified end-to-end to ensure PAIR does not alter protocol behavior. Per-endpoint timeout handling should also be tested by giving the DGX Spark a substantially longer first-token allowance while keeping shorter failure detection on the 5090 and 3090. Failover should be tested by stopping or saturating individual upstreams and confirming that requests move deterministically to the next eligible endpoint without disrupting healthy peers. Finally, validation should confirm that externally managed engine credentials and local runtime configuration never cross the PAIR trust boundary, are not exposed through discovery or telemetry, and are not written to logs. The reference implementation and current routing behavior used for comparison are available at https://github.com/darkmatter2222/HomeAILab/tree/main/router.

Confirmations

• I searched existing issues for duplicates.
• I agree to follow the Code of Conduct.

[dezfutak]

@darkmatter2222 - really love what you've developed & am toying with the idea of installing it, as I'm running hetergeneous hardware across a network; I stumbled across PAIR & then your comment/feature request.

The one question I have is: why would folks install PAIR instead of HomeLabAI - are you thinking that it's primarily a matter of ease-of-install leading to greater adoption maybe? I think in several ways HomeLabAI is superior, as it feels much more like a genuine heterogenous clustering-tool (mostly due to the restrictive shortcomings that you point out that PAIR suffers from).

[darkmatter2222]

@darkmatter2222 - really love what you've developed & am toying with the idea of installing it, as I'm running hetergeneous hardware across a network; I stumbled across PAIR & then your comment/feature request.

The one question I have is: why would folks install PAIR instead of HomeLabAI - are you thinking that it's primarily a matter of ease-of-install leading to greater adoption maybe? I think in several ways HomeLabAI is superior, as it feels much more like a genuine heterogenous clustering-tool (mostly due to the restrictive shortcomings that you point out that PAIR suffers from).

@dezfutak Thank you for the kind words. What I’ve built in HomeAILab is really a configuration that works specifically for my environment rather than something I designed as a broadly consumable product. I have a mix of GPUs scattered across different systems, and I use the router to make that heterogeneous setup behave predictably: deterministic routing, explicit GPU priority, capability filtering, capacity limits, and different runtime choices depending on the hardware. I think it’s useful as a concrete example of what this kind of system can look like, but I wouldn’t currently position it as a polished alternative that everyone should install.

That’s actually why PAIR is interesting to me. NVIDIA is already solving a lot of the difficult product-level problems around discovery, trust, node management, telemetry, UX, and general cluster operation. My router is more opinionated and currently more capable for my particular heterogeneous scheduling requirements, but I’m also the one responsible for maintaining all of it.

Now that I’ve seen PAIR, my view is that it’s only one or two important capabilities away from being a very effective solution for this exact use case. If they add a generic way to adopt externally managed OpenAI-compatible runtimes and provide stronger capability/capacity/priority-aware routing, I could potentially keep the tuned inference stack I want on each GPU while letting PAIR handle the cluster layer. At that point, I’d much rather use and contribute to a well-supported common platform than maintain a custom router indefinitely.

So I’d look at HomeAILab more as a working reference implementation of some of the behaviors I’d like to see PAIR support, rather than something I’m currently recommending people deploy instead of PAIR.

[jlacroix82]

Strong +1, with some empirical data that I think supports the core argument more
directly than expected: the Go services already do most of this today.

I spent yesterday testing whether an engine PAIR has never heard of can
participate in a cluster without patching anything. It can. On a 10-node DGX
Spark cluster (GB10 / aarch64), against stock PAIR 0.91.7 and engine-manager
0.17.4 with unmodified binaries, I registered a vllm engine purely by dropping
a manifest into <appdir>/engines/ — no fork, no rebuild.

mergeOntoBundled returns an override unchanged when the engine name isn't
bundled, so an unbundled engine is registered rather than rejected, and
ModelsResult iterates the whole registry. The result on the shipped binary:

loaded 3 engine manifest(s): [lmstudio ollama vllm]

Across two nodes over the pinned-mTLS control surface, in both directions:

x213 -> x220  GET  /v1/engines        vllm installed:true running:false
              POST /v1/engines/start  running:true healthy:true
              GET  /v1/models         modelsByEngine.vllm = ["LiquidAI/LFM2.5-350M"]
              POST /v1/models/load    -> loadedByEngine.vllm = ["LiquidAI/LFM2.5-350M"]
              POST /v1/models/unload, /v1/engines/stop  -> clean teardown, GPU freed
x220 -> x213  same engine list, remote start works

Real weights, real completions. So node inventory, trust, remote lifecycle and
model reporting are all already engine-agnostic. That seems worth knowing when
scoping this: the generic path may be much closer than "add support for each
runtime" implies.

Where it actually stops is the desktop app. With the engine installed,
running and healthy on two nodes and its model listed by both, the UI showed no
row for it on any node card — no error, no broken rendering, just absent.
EngineTypes is a closed literal union narrowed via isEngineType() at the
service-bridge boundary, and DispatcherBackend repeats the same two names, so
routing is closed too. #9 adds vLLM by extending that enumeration
(Extract<EngineType, 'ollama' | 'lm-studio' | 'vllm'>, PROXY_ENGINES), which
is exactly the per-engine cost this issue is about. The backend accepted my
engine; only the union kept it invisible.

Two Go-side blockers that a generic-backend design would still need to solve,
neither of which #9 touches:

1. actions.go selects the long-timeout client with a literal engine-name
   comparison:

   engineResponseHeaderTimeout     = 30 * time.Second
   ollamaLoadResponseHeaderTimeout = 10 * time.Minute
   ...
   if engine == "ollama" && action == "run_model" && e.ollamaLoadClient != nil {

   Every non-Ollama engine gets 30s for every action regardless of its manifest.
   A cold run_model against a just-loaded vLLM exceeded it in my testing until I
   pre-warmed the model. This looks like it wants to be a manifest field
   (actions.<name>.timeout_s).

2. ActionHTTP carries only method, path and body schema — no headers. Any
   engine behind auth therefore can't be driven by a manifest at all. Unsloth
   Studio, for example, returns 401 on /v1/models, so it can be probed for
   health but not managed. An optional headers map would cover it.

One more note relevant to the heterogeneous-routing part of this issue: engines
that bind one model per process need "restart on a different model" semantics.
#9's {model} placeholder plus engine:set-model handles that well for a
bundled engine. Before that existed I solved it with a small adapter that owns
the child process and presents a stable port, which also works for engines PAIR
doesn't bundle — reference implementation and the full test transcript here, in
case the adapter shape is useful for the arbitrary-backend case:
https://github.com/jlacroix82/pair-multi-engine

Happy to run experiments against real DGX Spark hardware if that would help
validate a design.

[darkmatter2222]

Strong +1, with some empirical data that I think supports the core argument more
directly than expected: the Go services already do most of this today.

I spent yesterday testing whether an engine PAIR has never heard of can
participate in a cluster without patching anything. It can. On a 10-node DGX
Spark cluster (GB10 / aarch64), against stock PAIR 0.91.7 and engine-manager
0.17.4 with unmodified binaries, I registered a vllm engine purely by dropping
a manifest into <appdir>/engines/ — no fork, no rebuild.

mergeOntoBundled returns an override unchanged when the engine name isn't
bundled, so an unbundled engine is registered rather than rejected, and
ModelsResult iterates the whole registry. The result on the shipped binary:

loaded 3 engine manifest(s): [lmstudio ollama vllm]

Across two nodes over the pinned-mTLS control surface, in both directions:

x213 -> x220  GET  /v1/engines        vllm installed:true running:false
              POST /v1/engines/start  running:true healthy:true
              GET  /v1/models         modelsByEngine.vllm = ["LiquidAI/LFM2.5-350M"]
              POST /v1/models/load    -> loadedByEngine.vllm = ["LiquidAI/LFM2.5-350M"]
              POST /v1/models/unload, /v1/engines/stop  -> clean teardown, GPU freed
x220 -> x213  same engine list, remote start works

Real weights, real completions. So node inventory, trust, remote lifecycle and
model reporting are all already engine-agnostic. That seems worth knowing when
scoping this: the generic path may be much closer than "add support for each
runtime" implies.

Where it actually stops is the desktop app. With the engine installed,
running and healthy on two nodes and its model listed by both, the UI showed no
row for it on any node card — no error, no broken rendering, just absent.
EngineTypes is a closed literal union narrowed via isEngineType() at the
service-bridge boundary, and DispatcherBackend repeats the same two names, so
routing is closed too. #9 adds vLLM by extending that enumeration
(Extract<EngineType, 'ollama' | 'lm-studio' | 'vllm'>, PROXY_ENGINES), which
is exactly the per-engine cost this issue is about. The backend accepted my
engine; only the union kept it invisible.

Two Go-side blockers that a generic-backend design would still need to solve,
neither of which #9 touches:

1. actions.go selects the long-timeout client with a literal engine-name
   comparison:

   engineResponseHeaderTimeout     = 30 * time.Second
   ollamaLoadResponseHeaderTimeout = 10 * time.Minute
   ...
   if engine == "ollama" && action == "run_model" && e.ollamaLoadClient != nil {

   Every non-Ollama engine gets 30s for every action regardless of its manifest.
   A cold run_model against a just-loaded vLLM exceeded it in my testing until I
   pre-warmed the model. This looks like it wants to be a manifest field
   (actions.<name>.timeout_s).

2. ActionHTTP carries only method, path and body schema — no headers. Any
   engine behind auth therefore can't be driven by a manifest at all. Unsloth
   Studio, for example, returns 401 on /v1/models, so it can be probed for
   health but not managed. An optional headers map would cover it.

One more note relevant to the heterogeneous-routing part of this issue: engines
that bind one model per process need "restart on a different model" semantics.
#9's {model} placeholder plus engine:set-model handles that well for a
bundled engine. Before that existed I solved it with a small adapter that owns
the child process and presents a stable port, which also works for engines PAIR
doesn't bundle — reference implementation and the full test transcript here, in
case the adapter shape is useful for the arbitrary-backend case:
https://github.com/jlacroix82/pair-multi-engine

Happy to run experiments against real DGX Spark hardware if that would help
validate a design.

@jlacroix82 this is extremely useful work — thank you for actually validating this against stock PAIR and real DGX Spark hardware rather than treating the engine abstraction as theoretical.

After going through https://github.com/jlacroix82/pair-multi-engine, I think your results make the case for this enhancement substantially stronger.

What stands out to me is that we may not be asking NVIDIA to redesign PAIR into a generic heterogeneous inference platform. A surprising amount of that architecture appears to already exist.

You demonstrated that an engine PAIR has never heard of can already be introduced declaratively, appear in the engine registry, participate in inventory, be controlled remotely across the pinned-mTLS surface, expose models, load/unload real weights, and operate across nodes — all using stock PAIR binaries. That is a very important distinction.

The remaining problem increasingly looks like opening and generalizing interfaces that are already there, rather than implementing a new architecture from scratch.

Your 10-node DGX Spark experiment and my much smaller but deliberately heterogeneous setup also expose two different dimensions of the same use case.

Your environment demonstrates horizontal scale:

"DGX Spark × 10 → multiple engines/models → one PAIR cluster"

Mine demonstrates heterogeneous specialization:

"RTX 5090 / vLLM → RTX 3090 / llama.cpp → DGX Spark / large VLM"

with different hardware architectures, runtimes, quantizations, throughput characteristics, context behavior, concurrency limits, and capabilities.

Those two cases together are why I think this is potentially much more significant than simply adding another supported engine.

If PAIR can make the engine boundary genuinely open and make endpoint capabilities/routing policy declarative, the same architecture starts working for clusters of two machines, ten machines, or potentially much larger deployments without every node needing to look alike.

For example, a real cluster might eventually look something like:

PAIR logical model / workload
|
+-- RTX 5090
| vLLM
| NVFP4 coding model
| text + tools
| priority 10
| capacity 4
|
+-- RTX 3090
| llama.cpp
| GGUF version of same logical model
| text + tools
| priority 20
| capacity 1
|
+-- DGX Spark
| SGLang / vLLM / specialized backend
| large MoE / long context
| priority 30
|
+-- DGX Spark
VLM backend
vision + text
dedicated vision capability

The critical point is that these endpoints are not interchangeable simply because they expose an OpenAI API.

They may differ in:

• GPU architecture
• VRAM or unified memory
• model architecture
• quantization
• inference engine
• maximum context
• prefill performance
• decode performance
• concurrency
• model-load time
• tool support
• vision capability
• reasoning support
• timeout requirements
• operational role

That means there are really two complementary pieces needed.

First, generic engine adoption: let an external runtime describe itself through a manifest instead of requiring PAIR source changes every time another backend appears.

Your work seems to demonstrate that much of this already functions in the Go services.

Second, capability-aware deterministic routing: once arbitrary engines can join the cluster, PAIR needs enough metadata to determine whether an endpoint is eligible for a particular request before applying normal load-based scheduling.

That distinction becomes extremely important with multimodal models.

If I send an image, PAIR should not ask:

«Which GPU is least busy?»

It should first ask:

«Which endpoints can actually satisfy this request?»

Something conceptually like:

request
↓
logical model
↓
text / vision requirement
↓
tool capability
↓
context requirement
↓
endpoint health
↓
configured admission capacity
↓
eligible candidates
↓
priority / scheduler

Only after those eligibility gates should GPU utilization, queue depth, warmness, or other dynamic scheduling signals determine placement.

The same applies to deterministic priority.

In my environment, I intentionally want:

RTX 5090
↓ full/unavailable
RTX 3090
↓ full/unavailable
DGX Spark

That isn't because the Spark is inferior hardware. It is because those devices have completely different performance envelopes for this particular workload.

For another model, the desired order could be exactly the opposite.

For vision, I may want:

vision request
↓
DGX Spark VLM only

For a very large model:

large-model alias
↓
DGX Spark pool

For a small high-throughput model:

small-model alias
↓
5090 pool

So ideally priority and capabilities belong to the endpoint/model relationship, not just globally to a machine.

This is where I think PAIR could become substantially more than a convenient way to connect a few similar PCs. It could become a genuinely useful local inference fabric across heterogeneous hardware.

The fact that your testing shows the backend already accepts an unbundled engine makes that possibility especially interesting.

Your two specific blockers are also exactly the kinds of things I would hope become manifest-driven rather than engine-name-driven:

actions..timeout_s

instead of special casing Ollama, and configurable local request headers/authentication rather than requiring another code path for every authenticated runtime.

Likewise, making the desktop consume the engine registry dynamically instead of maintaining a closed TypeScript union seems fundamental. Otherwise every new engine remains a PAIR release even though the backend has already demonstrated that it can accept it dynamically.

That is the architectural payoff I see here:

The PAIR binary should ideally understand capabilities and contracts, not an ever-growing list of inference-engine brand names.

Then first-class integrations can still exist for Ollama, LM Studio, vLLM, etc. and provide a better managed experience, while advanced deployments can bring SGLang, TensorRT-LLM, llama.cpp, Unsloth, custom OpenAI-compatible servers, containers, or whatever comes next without waiting for PAIR itself to add another enum value.

My current HomeAILab router implements the deterministic/capability side of this today:

https://github.com/darkmatter2222/HomeAILab/tree/main/router

and in particular the endpoint definitions:

https://github.com/darkmatter2222/HomeAILab/blob/main/router/config/router.yaml

and capability-aware selector:

https://github.com/darkmatter2222/HomeAILab/blob/main/router/internal/routing/classify.go

Your project demonstrates the other half — that PAIR's existing engine-management architecture is already much more generic than the product currently exposes:

https://github.com/jlacroix82/pair-multi-engine

Taken together, I think these are useful concrete reference implementations for what this enhancement could unlock.

@cguldogan — your work in #9 appears to solve an important piece of this for vLLM specifically, particularly "{model}" / "engine:set-model". I think the interesting question beyond #9 is whether that mechanism can ultimately become generic enough that the next vLLM/SGLang/TensorRT-LLM-style engine doesn't require another closed UI/routing enumeration.

@Noah-Tervalon-Nvidia — tagging you because this seems closely aligned with the direction described in PAIR's own README around making scheduling smarter for highly mixed hardware. Between a tested 10-node DGX Spark environment, the multi-engine work above, and the heterogeneous 5090/3090/GB10 deployment, there are now some concrete environments available to test this against rather than designing it abstractly.

If NVIDIA is interested in pursuing this direction, I suspect several of us in this thread would be willing to validate it against very different real hardware configurations.

To me, that's what makes this enhancement high leverage: it isn't just about adding vLLM. It potentially changes PAIR from a router for a defined collection of engines into a general heterogeneous local inference control plane, while preserving the discovery, trust, telemetry, UX, and cluster management NVIDIA has already built.

[darkmatter2222]

@dezfutak @jlacroix82 @cguldogan @Noah-Tervalon-Nvidia — pulling everyone into one place because I think the discussion here has converged on something much broader than a single vLLM feature.

Between the heterogeneous RTX 5090 / RTX 3090 / DGX Spark use case, @jlacroix82's testing across a 10-node DGX Spark environment, the generic-engine behavior that already appears to exist in PAIR's Go services, and the work underway in #9, I think there is now a fairly compelling case that PAIR is very close to becoming a genuinely useful heterogeneous inference control plane.

This matters well beyond one particular home-lab configuration. The same capability could apply across clusters of very different sizes and compositions:

• different NVIDIA GPU architectures and memory capacities
• different model architectures and quantizations
• vLLM, SGLang, llama.cpp, TensorRT-LLM and future backends
• small dense models and very large MoE models
• different context limits and concurrency characteristics
• text-only, tool-capable and vision/multimodal endpoints
• different prefill/TTFT and throughput characteristics
• specialist nodes versus general-purpose inference capacity

The piece I think is especially important is combining generic backend adoption with capability-aware, deterministic routing.

PAIR should first determine which endpoints are actually eligible for a request — logical model, vision capability, tools, context requirement, available capacity, health, etc. — and only then apply priority/load scheduling among those eligible endpoints.

That would allow deployments such as:

"RTX 5090 → RTX 3090 → DGX Spark"

for one workload, while a vision request might deterministically target a DGX Spark VLM pool, and a large-model request might route exclusively to high-memory nodes.

The important part is that the hardware does not need to be interchangeable, the models do not need to be identical, and the inference engines do not need to be identical. PAIR becomes the common cluster/control layer above all of them.

Between the work already demonstrated in:

https://github.com/jlacroix82/pair-multi-engine

and the heterogeneous routing reference implementation here:

https://github.com/darkmatter2222/HomeAILab/tree/main/router

it appears that a substantial portion of the underlying architecture is already within reach.

For everyone following or contributing to this discussion: if this is a direction you would actually use, please add a 👍, 🚀, or 👀 reaction to the original issue. That is probably a much cleaner signal to the NVIDIA PAIR team than filling the thread with repetitive "+1" comments, and it gives the maintainers a visible indication that this isn't an edge-case request from one deployment.

I think this enhancement could materially expand the audience for PAIR — from connecting a few similar local machines into something capable of orchestrating genuinely heterogeneous local AI infrastructure.

[jlacroix82]

I am fortunate (or not sometimes) to have basically unlimited access to random hardware so the paid app seems very useful as I haven't been able to come up with something myself that fully works. Happy to keep testing across a giant cluster of random devices.

[darkmatter2222]

I have Qwen 3.8 working on it right now

[dezfutak]

@darkmatter2222 - really love what you've developed & am toying with the idea of installing it, as I'm running hetergeneous hardware across a network; I stumbled across PAIR & then your comment/feature request.
The one question I have is: why would folks install PAIR instead of HomeLabAI - are you thinking that it's primarily a matter of ease-of-install leading to greater adoption maybe? I think in several ways HomeLabAI is superior, as it feels much more like a genuine heterogenous clustering-tool (mostly due to the restrictive shortcomings that you point out that PAIR suffers from).

@dezfutak Thank you for the kind words. What I’ve built in HomeAILab is really a configuration that works specifically for my environment rather than something I designed as a broadly consumable product. I have a mix of GPUs scattered across different systems, and I use the router to make that heterogeneous setup behave predictably: deterministic routing, explicit GPU priority, capability filtering, capacity limits, and different runtime choices depending on the hardware. I think it’s useful as a concrete example of what this kind of system can look like, but I wouldn’t currently position it as a polished alternative that everyone should install.

That’s actually why PAIR is interesting to me. NVIDIA is already solving a lot of the difficult product-level problems around discovery, trust, node management, telemetry, UX, and general cluster operation. My router is more opinionated and currently more capable for my particular heterogeneous scheduling requirements, but I’m also the one responsible for maintaining all of it.

Now that I’ve seen PAIR, my view is that it’s only one or two important capabilities away from being a very effective solution for this exact use case. If they add a generic way to adopt externally managed OpenAI-compatible runtimes and provide stronger capability/capacity/priority-aware routing, I could potentially keep the tuned inference stack I want on each GPU while letting PAIR handle the cluster layer. At that point, I’d much rather use and contribute to a well-supported common platform than maintain a custom router indefinitely.

So I’d look at HomeAILab more as a working reference implementation of some of the behaviors I’d like to see PAIR support, rather than something I’m currently recommending people deploy instead of PAIR.

That's really useful detail; just caught up with the other comments in this thread too. What a time for open source/localai :) My main interest is to be able to run a mix of older nvidia hardware too (pre-Turing) alongside Turing/Blackwell architecture. At the moment, PAIR wouldn't run on a larve VRAM Pascal card from its docs as far as I can tell.

[jlacroix82]

I am working on the UI a bit today as I have a weird setup… right now I have 9 x ThinkStation PGX devices, plus two gaming machines connected. I already have models running that I have set up (as seen below). Submitting a new PR with what I’ve done so far.

[jlacroix82]

[vonargo]

Hi folks! I just finished adding support for OpenAI-compatible endpoints in PAIR.

Hope this is useful!

[darkmatter2222]

Hi folks! I just finished adding support for OpenAI-compatible endpoints in PAIR.

Hope this is useful!

Thank you @vonargo Any chance you can add deterministic routing too?

[vonargo]

Thank you @vonargo Any chance you can add deterministic routing too?

@darkmatter2222 you're very welcome!

I opened a Q&A (#66) on that very subject. I think PAIR has quite the possibilities, but want to see where they intend to take the project.

I think this is definitely where it could go:

flowchart TD
    A[Application or workflow] --> B[PAIR]
    B -->C[Selects endpoint by capability fit]
    C --> D[Service endpoint A]
    C --> E[Service endpoint B]
    C --> F[Service endpoint C]

Loading

[jlacroix82]

@nv-pgoode @NV-sschneider — a direction question before anyone builds more here, grounded in a week of running the external-deployment policy from my branch on a 9-node fleet (8× DGX Spark in two TP=4 groups plus a gateway box).

What broke, and why it was always going to. Every PAIR failure I hit came from PAIR trusting a description of the fleet instead of looking at it:

• A TP=4 vLLM head disappeared from PAIR the moment its launcher renamed the container; the policy file named vllm-fn. The other group had the same problem on every relaunch because sparkrun gives each job a new suffix.
• An SGLang head was invisible until a hand-written port override, because the manifest assumes 30000 and the recipe uses 8888.
• Workers show Unknown unless someone types their container names into JSON on every node.
• Two probe paths were hard-coded to /health without knowing what /health costs on the engine behind it (see the bug I filed today).

Proposal: observe first, declare only to override.

1. Container-first discovery. engine-manager already runs docker container inspect for a named container. Make it an unnamed sweep every 15–30 s: list running containers, read image, command line and env, and recognise engines by what they execute (vllm serve, sglang.launch_server, llama-server, ollama). The command line already carries everything the policy asks the operator for: --port, --served-model-name, --tensor-parallel-size, --nnodes, --node-rank, --master-addr / --dist-init-addr. Names stop mattering.
2. Topology inference. rank 0 (or no rank) is the head; a rank whose master address matches another node's fabric IP is that node's worker. node-info already publishes each node's interfaces, so PAIR can map 10.0.1.18 to a node UUID itself. engine-deployments.json becomes an optional override, not the source of truth.
3. Bare-process fallback. Bare-host manual discovery already probes fixed ports for identity; widen it to listening sockets so a llama-server or vllm serve outside Docker is found the same way.
4. Identity endpoints are the only probes. Readiness and liveness use the manifest's identity endpoint; anything that generates is opt-in.
5. Observed vs declared as first-class state. Show what was detected, when, and how confident; route only to heads whose identity probe passes (already the rule), so detection never widens exposure. Auto-adopt behind a setting, or confirm once, to keep the "operator declaration is trusted" stance in the docs intact.

Phase 1 (the sweep replacing name lookup) alone would have prevented both of this week's outages of visibility. Phases 2–3 retire the mandatory policy file.

The question: is observed topology a direction you will accept in engine-manager, or do you want PAIR to remain the launcher of record and treat externally launched engines as out of scope? I will write it against develop if the answer is the former, and stop maintaining the policy-file approach if it is the latter. #64's adopt-by-URL and #66's routing discussion are the adjacent pieces.