Conversation
A node reachable only over an encrypted overlay may have no address a peer can assume. A Tailscale node's MagicDNS name outlives every literal it holds, and its IPv4 literal is a 100.64/10 CGNAT address that netpick already scores below a LAN address -- correctly, as a ranking. The exclusion was elsewhere: Candidates, RankRemote and IPsFromTXT all gated entries on net.ParseIP, so a node whose only address was a name produced an empty candidate list and Primary answered "". Every consumer had grown its own ad-hoc fallback for that, and they did not agree. Admit names through one shared test (Hostname / dialable) and score them between the public and private classes: a name re-resolves, so it survives the renumbering that strands a literal, but a LAN peer's own literal is still the better answer when both exist. The per-caller fallbacks that existed only to recover a .local name are now a second reading of the same source, so they collapse to the one case they still answer -- the node published no address at all and only its discovery host name is left. CGNAT, IPv6 ULA and virtual-adapter demotion stay rankings. Nothing here promises reachability; a consumer that must connect still confirms by connecting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
vLLM serves exactly one model per process, so its launch command names
the model. Add a {model} placeholder resolved from a new runtime.model,
a runtime.extra_args tail appended to runtime.args, and the persistent
engine:set-model / one-shot engine:start {model} setters that choose it.
Starting an engine whose launch template needs {model} with none
configured now fails with guidance instead of spawning.
The bundled vllm.json is Linux-only (vLLM ships no Windows or macOS GPU
build), process mode, port 8000, /health readiness with a 30-minute
budget for the first weight download, and the OpenAI list/loaded/chat
actions. Adoption is the primary path: process mode reconciles an
already-serving listener even when detect misses, so a user's own
`vllm serve` or the vllm/vllm-openai container is picked up.
engine:set-port and engine:set-model share one override file, so both
now read-modify-write it rather than replacing it with a single-key
delta, and the file is unlinked only once no override remains.
allowedPlaceholders had been doing double duty as the set an
engine:action caller may not supply. Split out
reservedActionPlaceholders so {model} stays caller-suppliable on a
model action while remaining a valid manifest placeholder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
A node's {service: port} set existed in exactly one place a peer could read
it: the ni=/ol=/lm=/em=/ec=/... keys on its mDNS record. Multicast does not
cross a routed or overlay network, so a peer on a Tailscale tailnet can
learn that a node exists -- someone typed its address -- and nothing about
what it runs. It cannot tell a PAIR node from a bare Ollama box, and it has
no way to find the node's engine manager or its promoted proxy ports.
node-info is the one inter-node surface deliberately kept plain, which
makes it the one place such a peer can ask. Carry the set there.
The broker owns the set (it assigns and re-assigns those ports) and already
holds it as the registration cache it replays to the scanner. It now
projects that same cache into a nodeinfo:set-services push, on node-info
spawn and on every register/unregister, so the HTTP answer and the mDNS
record are one derivation rather than two.
The set is always sent whole: a service that stopped is expressed by its
key being absent, exactly as an unregister is on the record. Absent from
the response entirely means the parent has not pushed yet -- not "this node
runs nothing", which a peer would act on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Adding a PAIR node by address did not work, and could not have. The prober asked its 11434 and 1234 in plaintext, but on a PAIR node those ports carry the proxy facades, which refuse plaintext from anything but loopback -- so every probe answered 403 and the peer read as having no engines. If it had been bridged anyway, the proxies would have dialed it plain (a manual candidate carries no cluster principal, so HasPin is false and routing falls to the manual arm) into the same refusal. Manual nodes only ever worked for the bare Ollama / LM Studio box they were built for. That box still exists and is unchanged. What is new is the second kind of manual node: a peer that is a PAIR node and is simply on the far side of a network that carries no multicast, such as a Tailscale tailnet. It is not a special case, it is an ordinary peer with no discovery between here and there, so it is made into one. - The prober asks node-info FIRST, and its answer decides everything else. A service map identifies a PAIR node, and such a node is never probed on its engine ports again. - A PAIR node's models come from its engine manager over pinned mTLS, the same fetch the scanner makes for a discovered peer. No pin, no models: before pairing the node still appears with its hardware. - The broker synthesizes the directory record the scanner would have produced and pushes it into the discovery relay, so both proxies, the scheduler, engine-manager's remote operations, the workload relay and the errors peer sync all see a pinned peer. It carries the peer's cluster principal, which is what gets it dialed over mTLS instead of 403'd. - It refuses to synthesize a record for this host itself (by identity, so the overlay name for this machine is caught too) or for a node the scanner already owns, and it withdraws only records it put there. - The raw-engine bridge is withdrawn for a PAIR node rather than left alongside, so there is one route to a node and not two. node/add also gains per-service port overrides, and now rejects a "host:port" address with the reason instead of accepting an entry that can never be reached: service ports are appended to that value. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
vLLM speaks the same OpenAI HTTP surface as LM Studio, so it needs the same router rather than a new binary: lmstudio-proxy becomes the OpenAI-compatible proxy for every such engine. It subscribes to lm and vl, each routing candidate records which engine it represents, and workloads are tagged with that engine instead of a package constant. A peer advertises lm and vl at the same value — its one proxy port — so a dual-engine peer still projects to a single routing target whose per-engine model attribution says which engine owns what. A manual node is the exception: the user supplies an engine's own address and LM Studio and vLLM sit on different ports, so node/add-manual and node/remove-manual now carry an engine and the manual overlay is keyed by (engine, node). The reachability cache is keyed the same way, while scheduledOn and node selection stay keyed by the bare node id. node/set-local-backend becomes per-engine, so one node may run both engines at once and clearing one leaves the other routing. Cluster ingress picks the local engine that actually serves the requested model — read from the engines themselves, since this node's own discovery entry can lag — and answers GET /v1/models by merging both local engines, which a single forward would have hidden half of. Broker: runAutoAdvertiseVLLM registers vl at the OpenAI proxy's port and hands the engine's loopback port to that proxy. There is no managed facade for vLLM, so the loop carries none of LM Studio's facade handling. Manual nodes gain a vllm_* triple probed on 8000, with vLLM's own /version as the disambiguator from any other OpenAI server. The scheduler ranks vllm alongside the others, and the TUI's second proxy panel now names both engines it fronts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Node A is a real broker with its real workers; node B is a real cluster-manager and a real ollama-proxy in front of a fake engine, plus the two HTTP surfaces a peer exposes. A pairs with B over the real PIN exchange addressed by nothing but an address and a port -- no nodeId, because A never discovered B and never will -- and is then told about it with one node/add. It asserts the three things "behaves like a discovered peer" has to mean: B appears in A's directory as a trusted, clustered peer with the model list A read from B's engine manager over cluster mTLS; A's proxy routes an inference request to B over cluster mTLS and B's engine serves it; and A opens zero plaintext connections to B's engine ports, which is the regression this change exists to fix. Two real brokers cannot share one loopback -- every broker-owned port is a compiled-in constant -- so B is a stub peer on ephemeral ports, reached through node/add's new per-service port overrides. Everything the assertions turn on is real: the identities, the pins, the pairing, the proxy, and the transport choice. Also adds the PairNode hysteresis the test implies. Recomputing it per probe meant one missed node-info answer -- three seconds, routine across an overlay network -- probed the peer's proxy facades in plaintext, blanked its service map and withdrew it from every consumer for a cycle. It now holds across a failure episode on the same counter discovery uses, and reverts past it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Add node only ever sent an invite. On a LAN that was enough, because discovery brought the peer in anyway. On a network that carries no multicast it is not: nothing writes manual-nodes.json, no record ever arrives, and a peer could pair successfully and stay invisible forever. The dialog now records the address as a manual node and tells the broker about it before inviting. That order is the useful one: the node appears with its hardware as soon as it answers, which is the only feedback an operator gets that a hand-typed address is right, and a pairing that fails leaves something on screen to retry against instead of nothing. The field is an address or a host name, with the guidance a VPN user needs -- use the MagicDNS name, it re-resolves -- and a Service ports disclosure for a node whose services do not sit on the defaults. The overrides are persisted with the entry, because the replay after a restart is the only thing that re-creates it: without them a node on non-default ports comes back unreachable. The two spellings of those ports (camelCase here, snake_case on the wire) meet in one projection, narrowed rather than cast. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
A new page for the case the rest of the documentation did not cover: two machines that can reach each other but have no local link between them. What a tailnet changes (no discovery, CGNAT addresses, MagicDNS names), the add-by-name → pair → verify walkthrough, a Tailscale ACL grant that opens exactly the eight ports PAIR needs between tagged nodes and says what each is for, and the four failures that are specific to this setup -- node with no models, 403 from a peer, a name that will not resolve, and a node that was found and went quiet. Linked from the README reading order, from the empty-discovery section of Troubleshooting (where an empty list is the expected result, not a fault), and from the Manual Nodes section of Architecture, which now describes the two kinds of manual node rather than one. The README and SECURITY.md said prompts remain on the local network. They now say the network you route them over -- your local network, or an encrypted overlay you configured -- which is what was always true and is now reachable. SECURITY.md's local-network boundary section says what an overlay changes about it: the encryption and the pinning are the same, but everyone admitted to the overlay is on that boundary. Versions: MINOR for node-info, manual-nodes, ui-broker and both proxies -- each gained behavior visible over IPC or HTTP, and a node reachable only by name is newly routable. PATCH for scanner, errors, workload-manager and cluster-manager, which only pick up the netpick change. Product and installer MINOR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Add the 'vllm' engine type, display name, links, capabilities and icon, and offer it in onboarding on Linux only, never pre-selected. A node reported by the OpenAI-compatible proxy is no longer necessarily LM Studio, so the bridge stops inferring the engine from the proxy's name: a source that fronts one engine still names it, while the OpenAI proxy's nodes carry per-engine model attribution and that is what says whether a node runs LM Studio, vLLM, or both. The proxy's ready port and its manual-node bridge now apply to every engine it fronts, and those bridge calls name their engine so two engines on one host cannot collide. vLLM serves one model per process, chosen before it starts, so engine settings gains a "Model to serve" field behind a new hasServedModel capability. It relays the new engine:set-model, and EngineStatus gains the configured model so the field can render what is actually set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The synthesized directory record is shape-identical to a discovered one, so the renderer needs no special case for it -- which is exactly the property worth a regression gate, because it is invisible until it breaks. Drives the bridge state with the record the broker synthesizes for a peer added by address, and asserts the whole surface a node card is built from: the card itself, trust and membership, per-engine models with their loaded state, the node-info poll target, and the hardware that comes back from it. The address is a MagicDNS name throughout, since that is the part that is new: nothing on this path may require an IP literal, and an empty address is what a node with nowhere to be dialed looks like. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The broker's cluster-manager port is a compiled-in constant, so only one broker on a machine can hold it, and a previous test's broker can still be tearing its workers down when the next one starts. When that happened this test's own cluster-manager silently failed to bind and the pairing completion was posted to whichever process did hold the port, which read as a failed pairing. Wait for the port to be released before starting, and for this broker's own listener to exist before inviting, instead of sleeping a second and hoping. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
README, overview, architecture and the per-service READMEs now name three engines, and engine-lifecycle gains a vLLM section covering what makes it different: Linux only, adoption of an instance you already run as the normal path, one model per process chosen before start, a first start that can take many minutes, and no model operations. lmstudio-proxy's README is rewritten as the OpenAI-compatible proxy that fronts both LM Studio and vLLM, keeping the binary name as the wire contract it is. Versions: lmstudio-proxy MAJOR (node/add-manual and node/remove-manual now require an engine); engine-manager, ui-broker, manual-nodes and job-scheduler MINOR (additive IPC); node-scanner, errors, workload-manager and tui PATCH (recompiled for the shared discovery key, plus the TUI's panel label). node-info, node-settings and cluster-manager are unchanged. Product and installer MINOR for the new engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The engine manager writes manifest overrides for its own host, and there is no remote served-model control on the wire, so a peer's row could never hold a value. Gate the row (and the accordion's "Ports and model" title) on the local node, and drop the now-unreachable read-only branch from the row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The parity and backend docs, and the READMEs of the broker, scheduler, TUI,
and node-scanner, all enumerate engines or service keys by hand and nothing
in the contract checker notices a missing engine. Add vLLM to each: the
routing row, the vl discovery key, the three advertise loops, the per-engine
schedule:priority list, the OpenAI-compatible proxy's two engines, the
per-engine node/set-local-backend, and the fact that vLLM has no model
operations at all because the model is a start-time setting.
Also fix a mangled sentence in the manual-nodes README ("The engine engine
ports are compiled in") introduced when the probe list grew a fourth leg.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Combine vLLM engine support with Tailscale-reachable manual nodes. Conflicts resolved in nvpair-manual-nodes (vLLM becomes a third engine leg of the bare-host probe, keyed by the per-service port overrides, and is never probed on a PAIR peer) and in versions.json (higher of each component's bump kept). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
SGLang serves the same OpenAI HTTP surface as vLLM and, like vLLM, one
model per process named on the command line, so its manifest is the vLLM
one with SGLang's own facts: the `sglang serve --model-path {model}`
launch, stock port 30000, /health readiness with the same 30-minute
first-start budget, an install into a uv venv of `sglang[all]`, and
Linux only. Adoption of an instance the user already runs (a
`sglang serve` or the lmsysorg/sglang container) is the primary path,
exactly as for vLLM.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
(cherry picked from commit 5cd6185)
Four files have carried misaligned const, struct and field blocks since a comment or a field was inserted into an already-aligned group without re-running gofmt. Every one of them sits on or beside the SGLang change, so fixing them first keeps that diff about SGLang rather than about whitespace a formatter would have moved anyway. Whitespace only: the compiled output is byte-identical, so no component version moves. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit 24d6077)
A tensor-parallel vLLM group (--nnodes N) has one API on its head node and headless workers that open no port. Document that PAIR adopts it on the head only, that a worker correctly shows no engine, and that a worker must never be restarted alone. PAIR does not combine GPUs or split a model across nodes; it sees the group vLLM formed as one engine on that group's head node. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit 3697089)
SGLang speaks the same OpenAI HTTP surface as LM Studio and vLLM, so it joins them behind the one router rather than getting a binary of its own: the proxy's engine table gains a third entry, it subscribes to sg as well as lm and vl, and everything keyed by engine — routing candidates, model attribution, workload tags, local backends, the manual overlay — carries sglang without any new mechanism. A peer advertises sg at the same value as lm and vl, its one proxy port, so a three-engine peer still projects to a single routing target. The resolution order is lmstudio, vllm, sglang, which is the deterministic tie-break when more than one engine on a node claims the same model id. Broker: runAutoAdvertiseSGLang registers sg at the OpenAI proxy's port and hands the engine's loopback port to that proxy, keyed by engine so it never disturbs the other two. Like vLLM there is no managed facade, so the loop carries none of LM Studio's facade handling. /health is the liveness probe and is a stronger signal here than elsewhere: SGLang binds its port only once the model is loaded, so a listening /health means ready to serve. Manual nodes gain an sglang_* triple probed on 30000, disambiguated by GET /get_model_info — a route neither LM Studio nor vLLM serves. probeVLLM now refuses a server that answers it: /get_model_info is SGLang's own route, so a server answering it is SGLang whatever else it serves, and vLLM must not claim it. That is defense in depth rather than an observed collision — SGLang answers no /version, so it never reaches the check — and an error on /get_model_info reads as "not SGLang, carry on", so a slow vLLM cannot flap to down. Model ids get no interpretation anywhere. SGLang's /v1/models id is its --served-model-name, which defaults to --model-path verbatim and is routinely a bare local directory rather than a Hugging Face id. The scheduler ranks sglang alongside the others, and the TUI's second proxy panel names all three engines it fronts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit 63aeb0c)
Every services README that enumerates engines or discovery keys by hand now names SGLang: the proxy's three fronted engines and its lm/vl/sg subscription, the sg key on the node-scanner's record, the broker's fourth advertise loop and per-engine manual bridge, the scheduler's fourth priority contract, the manual-node sglang_* triple and its probe leg, the engine-manager's second one-model-per-process engine, and the TUI panel that fronts all three. Two things the vLLM pass left behind are corrected while the same paragraphs are open: the proxy README still said a forwarded workload's engine "is always lmstudio", which stopped being true when vLLM landed; and the model-id shapes are now stated outright, because SGLang's --served-model-name defaults to --model-path verbatim and so is routinely a bare local directory rather than a Hugging Face id. Nothing in the tree parses a model id, and the READMEs now say so. The manual-nodes README describes the disambiguation from both sides: /version identifies vLLM, /get_model_info identifies SGLang, and the vLLM probe refuses a server that answers the latter because that route is SGLang's own. Versions: lmstudio-proxy MINOR (node/add-manual accepts a third engine and the subscription gains sg — additive, the engine field was already required); ui-broker, manual-nodes (sglang_* status fields and an sglang port override), job-scheduler and engine-manager MINOR — the last for the SGLang manifest bundled in the preceding commit, which shipped without a bump of its own. node-scanner, errors and workload-manager PATCH (recompiled against the shared discovery key) and tui PATCH (the panel label). ollama-proxy, node-info, node-settings and cluster-manager carry no source change and keep their versions, matching how the vLLM pass treated them; the first three of those do link noderec and so are technically recompiled, but a new unreferenced service key changes nothing a reader of --version would learn. Product and installer MINOR for the new engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit bc28905)
Add the 'sglang' engine type, display name, links, capabilities and icon, and offer it in onboarding on Linux only, never pre-selected. The engine-manager id and our EngineType are the same string, so every name translation is a pass-through; LM Studio stays the only engine whose two spellings differ. SGLang is another OpenAI-compatible server behind the same proxy as LM Studio and vLLM, so it joins PROXY_SOURCE_ENGINES['lmstudio-proxy'] and takes its presence from the per-engine model attribution that proxy already stamps on every node it reports. PROXY_REPRESENTATIVE_ENGINES is deliberately unchanged: it holds one entry per proxy *process*, not per engine. Like vLLM it serves one model per process, chosen before it starts and resident until it stops, so it declares hasServedModel and no model operations at all. Its /v1/models id is the --served-model-name, which defaults to --model-path verbatim: either a Hugging Face repo id or a local model directory. formatModelDisplayName grew a rule for that second shape, because the Hugging Face formatter renders /models/my-model as "models/my model"; a path now shows its last segment unhumanised, since a directory name is only recognisable as the operator typed it. The rule sits in the shared default branch, so vLLM — which takes local paths too — gets it as well, while the engines with formatting rules of their own are untouched. It defers to the 256-char input cap: that cap slices from the right, exactly where a path keeps its last segment, so a capped path is handed back truncated rather than named after a truncated middle segment. ManualServicePorts gains sglang (stock port 30000) alongside vllm, and the stale "carried but not probed yet" note on that field goes with it. SGLang's logo is not redistributable, so its tile is drawn the way vLLM's is: "SG" in white on #c2410c. Orange rather than another blue — the two tiles sit next to each other in the backend list, and blue-against-violet is the pair the common colour-vision deficiencies collapse. White on #c2410c is 5.2:1, past the 4.5:1 floor. desktop/docs/services-api.md needed no change: it is generated by verify-service-contracts.ts and enumerates JSON-RPC methods, not engines. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit 99c421f)
SGLang joins Ollama, LM Studio and vLLM as a fourth engine, and every place the docs enumerate engines or list engine ports does it by hand, so nothing catches an engine that was added to the code but not the prose. Add it to each: the README's engines row and its two later mentions; the overview's engine definition and proxy sentence; architecture's diagram label, engine definition, model-id matching, one-pool note, install-location sentence and port table; and the engine-lifecycle engine lists, adoption sentence and adopted-port-change bullets. engine-lifecycle gains a full SGLang section next to vLLM's, covering what is specific to it: stock port 30000 behind the same OpenAI-compatible proxy, a port that opens only once the model has finished loading so a loading node shows no engine at all, GET /health for readiness and liveness, GET /get_model_info as what tells a bare SGLang host apart from any other OpenAI server, a model directory as well as a Hugging Face id under Model to serve, and --served-model-name defaulting to --model-path verbatim, which is the id a user actually sees in Endpoints. A multi-node subsection mirrors vLLM's: --nnodes with --node-rank and --dist-init-addr, adopted on the head only, worker ranks correctly showing no engine, and never restarting one alone. Both multi-node sections say plainly that the sharding is the engine's own and that PAIR neither combines GPUs nor splits a model across nodes; it sees the group as one engine on its head node. getting-started, terminal-interface and troubleshooting enumerate engines too, but the vLLM commits deliberately left them alone; naming SGLang in a page that has never heard of vLLM would read worse than leaving it, so they are untouched here as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit 84c6b98)
On current SGLang builds GET /health is not a liveness check: it runs a real forward pass and answers in almost exactly one second (measured 1.001 s, the same as /health_generate), where /get_model_info and /v1/models answer in under a millisecond. PAIR probes an engine's ready route constantly — every engine:status from the broker's advertise loop, every model-list request from every peer's scanner, the loaded-model watcher, and the health loop — and each of those holds the engine's operation mutex for the probe. On a node with several peers that adds up to more than a second of mutex demand per second, so the model endpoint serialized and then never drained: peers' requests piled up in CLOSE-WAIT, the engine-manager's descriptor count climbed past 240, and the node's own proxy answered 502 for the model it was serving, while a peer (which had fetched the list once before the queue formed) routed to it fine. Point every SGLang probe at /get_model_info instead: the manifest's ready and health probes, and the broker's checkSGLangHealth. It is instant, it is SGLang's own route — so, unlike /health, which vLLM also serves, a 200 positively identifies the engine — and because SGLang binds its port only once the model is loaded, a 200 still means ready to serve. The user doc says why /health is avoided so nobody "fixes" it back. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com> (cherry picked from commit b1ef53b)
Collaborator
|
Hey, thanks for adding this! We don't have a direction right now on adding SGLang, but will keep you posted. |
|
One hazard to fold in from running this on a 4-node SGLang head for a week: on SGLang,
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PAIR routes to Ollama, LM Studio and (with #9) vLLM. This adds SGLang as a fourth engine on the same footing: detected or adopted, started and stopped, advertised over discovery, routed to, scheduled, and shown in the desktop and terminal interfaces.
User-visible outcome:
30000(asglang serve --model-path …or thelmsysorg/sglangcontainer) is adopted rather than restarted. That is the normal path.--served-model-name, which defaults to the--model-pathverbatim, so a local directory shows up under its own path. There are no pull, load, eject or delete operations./v1/models, each request routed to the local engine that owns the requested model and each workload tagged with that engine.30000, told apart from other OpenAI-compatible servers byGET /get_model_info, a route only SGLang serves. The scheduler ranks SGLang nodes with the same node-wide load signal as the other engines.--nnodes N) is adopted on its head rank only; worker ranks open no port and correctly show no engine. The docs say so, next to the equivalent vLLM note.One SGLang-specific finding worth knowing: on current SGLang builds
GET /healthruns a real forward pass and takes about a second (the same as/health_generate), while/get_model_infoanswers in well under a millisecond. PAIR probes an engine's ready route often (status polls, model-list requests from every peer, the loaded-model watcher, the health loop), so every SGLang probe here uses/get_model_info. With/healtha node with several peers serialized behind the engine's operation mutex and its model endpoint stopped answering; the last commit explains the measurement.Related: #24 asks for adoption of OpenAI-compatible backends and names SGLang as one; this covers the SGLang case the way #9 covers vLLM, through the one shared OpenAI-compatible proxy rather than a new binary.
Scope
Included: engine manifest (
services/nvpair-engine-manager/manifests/sglang.json, Linux amd64/arm64,uvvenv install ofsglang[all]), discovery keysg, the OpenAI-compatible proxy's engine table (resolution order LM Studio, vLLM, SGLang), broker advertise loop and manual-node bridge, the bare-host probe leg with its/get_model_infodisambiguator (and the matching guard so the vLLM leg never claims an SGLang), scheduler engine list, TUI label, desktop engine type, capabilities, icon and per-engine attribution, readable display of path-like model ids, docs, and tests for each of those.Excluded: a managed port facade for SGLang (it keeps
30000and is never moved), model download or deletion, any Windows or macOS install path, and authentication headers for an SGLang behind a token (#26).Stacked on #9 and #10. SGLang needs #9's OpenAI-proxy generalization and served-model machinery, and #10's per-service port overrides for manually added nodes, so this branch is based on the merge of those two. Until they land, the diff here shows all three; the SGLang-only change is
https://github.com/cguldogan/Personal-AI-Router/compare/d676885...pr/sglang-engine(8 commits, 61 files). No breaking IPC change:node/add-manualandnode/remove-manualsimply acceptsglangas a thirdenginevalue.Validation
Environment: macOS 15 arm64, Go 1.27, Node 23; two DGX Spark (GB10, aarch64) nodes running a real two-rank SGLang group as the engine under test.
All pass: gofmt clean and vet clean in every module; 7 of the 8 Go modules fully green; the interop suite passes with 0 skips; desktop typecheck clean, lint 0 errors (one pre-existing prettier warning in a file this change does not touch), 243 unit tests in 42 files pass; service contracts and dead-code checks pass; every commit is signed off. The one failure is
nvpair-engine-manager TestUninstallTerminatesRunningInstance, which fails identically on the unmodified base tree on this host (it cannot resolve the image path of the process holding a port; the same failure is noted in #9) and is unrelated.A note for reviewers reading the series in order: the docs commit says SGLang readiness uses
GET /health, and the final commit changes that toGET /get_model_infoafter measuring the cost against a real instance. Each message is accurate for its own commit; the end state is/get_model_infoeverywhere.End-to-end against a real SGLang (
lmsysorg/sglang,--tp 2 --nnodes 2, head rank on30000):sgat its proxy port, its/v1/modelscarries the model attributed tosglang, and a chat completion sent to a peer's PAIR proxy routes over the cluster to the head node and answers./get_model_infochange; with/healththey took 1 s each and serialized.Screenshots of the node card (SGLang adopted on the head node) and of the Jobs panel (a run tagged SGLang) follow in a comment.
Risk
enginevalue, newsglang_*manual-node status fields and asglangport override. Consumers that ignore unknown keys and fields are unaffected; the desktop is updated in the same change./healthreintroduces the one-second-per-probe cost described above. The manifest, the broker probe and the user doc all say why.Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.json, and described user-visible changes above so they reach the release notes.