Conversation
A node reachable only over an encrypted overlay may have no address a peer can assume. A Tailscale node's MagicDNS name outlives every literal it holds, and its IPv4 literal is a 100.64/10 CGNAT address that netpick already scores below a LAN address -- correctly, as a ranking. The exclusion was elsewhere: Candidates, RankRemote and IPsFromTXT all gated entries on net.ParseIP, so a node whose only address was a name produced an empty candidate list and Primary answered "". Every consumer had grown its own ad-hoc fallback for that, and they did not agree. Admit names through one shared test (Hostname / dialable) and score them between the public and private classes: a name re-resolves, so it survives the renumbering that strands a literal, but a LAN peer's own literal is still the better answer when both exist. The per-caller fallbacks that existed only to recover a .local name are now a second reading of the same source, so they collapse to the one case they still answer -- the node published no address at all and only its discovery host name is left. CGNAT, IPv6 ULA and virtual-adapter demotion stay rankings. Nothing here promises reachability; a consumer that must connect still confirms by connecting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
A node's {service: port} set existed in exactly one place a peer could read
it: the ni=/ol=/lm=/em=/ec=/... keys on its mDNS record. Multicast does not
cross a routed or overlay network, so a peer on a Tailscale tailnet can
learn that a node exists -- someone typed its address -- and nothing about
what it runs. It cannot tell a PAIR node from a bare Ollama box, and it has
no way to find the node's engine manager or its promoted proxy ports.
node-info is the one inter-node surface deliberately kept plain, which
makes it the one place such a peer can ask. Carry the set there.
The broker owns the set (it assigns and re-assigns those ports) and already
holds it as the registration cache it replays to the scanner. It now
projects that same cache into a nodeinfo:set-services push, on node-info
spawn and on every register/unregister, so the HTTP answer and the mDNS
record are one derivation rather than two.
The set is always sent whole: a service that stopped is expressed by its
key being absent, exactly as an unregister is on the record. Absent from
the response entirely means the parent has not pushed yet -- not "this node
runs nothing", which a peer would act on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Adding a PAIR node by address did not work, and could not have. The prober asked its 11434 and 1234 in plaintext, but on a PAIR node those ports carry the proxy facades, which refuse plaintext from anything but loopback -- so every probe answered 403 and the peer read as having no engines. If it had been bridged anyway, the proxies would have dialed it plain (a manual candidate carries no cluster principal, so HasPin is false and routing falls to the manual arm) into the same refusal. Manual nodes only ever worked for the bare Ollama / LM Studio box they were built for. That box still exists and is unchanged. What is new is the second kind of manual node: a peer that is a PAIR node and is simply on the far side of a network that carries no multicast, such as a Tailscale tailnet. It is not a special case, it is an ordinary peer with no discovery between here and there, so it is made into one. - The prober asks node-info FIRST, and its answer decides everything else. A service map identifies a PAIR node, and such a node is never probed on its engine ports again. - A PAIR node's models come from its engine manager over pinned mTLS, the same fetch the scanner makes for a discovered peer. No pin, no models: before pairing the node still appears with its hardware. - The broker synthesizes the directory record the scanner would have produced and pushes it into the discovery relay, so both proxies, the scheduler, engine-manager's remote operations, the workload relay and the errors peer sync all see a pinned peer. It carries the peer's cluster principal, which is what gets it dialed over mTLS instead of 403'd. - It refuses to synthesize a record for this host itself (by identity, so the overlay name for this machine is caught too) or for a node the scanner already owns, and it withdraws only records it put there. - The raw-engine bridge is withdrawn for a PAIR node rather than left alongside, so there is one route to a node and not two. node/add also gains per-service port overrides, and now rejects a "host:port" address with the reason instead of accepting an entry that can never be reached: service ports are appended to that value. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Node A is a real broker with its real workers; node B is a real cluster-manager and a real ollama-proxy in front of a fake engine, plus the two HTTP surfaces a peer exposes. A pairs with B over the real PIN exchange addressed by nothing but an address and a port -- no nodeId, because A never discovered B and never will -- and is then told about it with one node/add. It asserts the three things "behaves like a discovered peer" has to mean: B appears in A's directory as a trusted, clustered peer with the model list A read from B's engine manager over cluster mTLS; A's proxy routes an inference request to B over cluster mTLS and B's engine serves it; and A opens zero plaintext connections to B's engine ports, which is the regression this change exists to fix. Two real brokers cannot share one loopback -- every broker-owned port is a compiled-in constant -- so B is a stub peer on ephemeral ports, reached through node/add's new per-service port overrides. Everything the assertions turn on is real: the identities, the pins, the pairing, the proxy, and the transport choice. Also adds the PairNode hysteresis the test implies. Recomputing it per probe meant one missed node-info answer -- three seconds, routine across an overlay network -- probed the peer's proxy facades in plaintext, blanked its service map and withdrew it from every consumer for a cycle. It now holds across a failure episode on the same counter discovery uses, and reverts past it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
Add node only ever sent an invite. On a LAN that was enough, because discovery brought the peer in anyway. On a network that carries no multicast it is not: nothing writes manual-nodes.json, no record ever arrives, and a peer could pair successfully and stay invisible forever. The dialog now records the address as a manual node and tells the broker about it before inviting. That order is the useful one: the node appears with its hardware as soon as it answers, which is the only feedback an operator gets that a hand-typed address is right, and a pairing that fails leaves something on screen to retry against instead of nothing. The field is an address or a host name, with the guidance a VPN user needs -- use the MagicDNS name, it re-resolves -- and a Service ports disclosure for a node whose services do not sit on the defaults. The overrides are persisted with the entry, because the replay after a restart is the only thing that re-creates it: without them a node on non-default ports comes back unreachable. The two spellings of those ports (camelCase here, snake_case on the wire) meet in one projection, narrowed rather than cast. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
A new page for the case the rest of the documentation did not cover: two machines that can reach each other but have no local link between them. What a tailnet changes (no discovery, CGNAT addresses, MagicDNS names), the add-by-name → pair → verify walkthrough, a Tailscale ACL grant that opens exactly the eight ports PAIR needs between tagged nodes and says what each is for, and the four failures that are specific to this setup -- node with no models, 403 from a peer, a name that will not resolve, and a node that was found and went quiet. Linked from the README reading order, from the empty-discovery section of Troubleshooting (where an empty list is the expected result, not a fault), and from the Manual Nodes section of Architecture, which now describes the two kinds of manual node rather than one. The README and SECURITY.md said prompts remain on the local network. They now say the network you route them over -- your local network, or an encrypted overlay you configured -- which is what was always true and is now reachable. SECURITY.md's local-network boundary section says what an overlay changes about it: the encryption and the pinning are the same, but everyone admitted to the overlay is on that boundary. Versions: MINOR for node-info, manual-nodes, ui-broker and both proxies -- each gained behavior visible over IPC or HTTP, and a node reachable only by name is newly routable. PATCH for scanner, errors, workload-manager and cluster-manager, which only pick up the netpick change. Product and installer MINOR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The synthesized directory record is shape-identical to a discovered one, so the renderer needs no special case for it -- which is exactly the property worth a regression gate, because it is invisible until it breaks. Drives the bridge state with the record the broker synthesizes for a peer added by address, and asserts the whole surface a node card is built from: the card itself, trust and membership, per-engine models with their loaded state, the node-info poll target, and the hardware that comes back from it. The address is a MagicDNS name throughout, since that is the part that is new: nothing on this path may require an IP literal, and an empty address is what a node with nowhere to be dialed looks like. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The broker's cluster-manager port is a compiled-in constant, so only one broker on a machine can hold it, and a previous test's broker can still be tearing its workers down when the next one starts. When that happened this test's own cluster-manager silently failed to bind and the pairing completion was posted to whichever process did hold the port, which read as a failed pairing. Wait for the port to be released before starting, and for this broker's own listener to exist before inviting, instead of sleeping a second and hoping. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
This was referenced Sep 4, 2026
|
Tested the mixed-version case, since that is what anyone adopting this will live in for a while: one node on this branch, its peers on 0.1.1. Environment. All thirteen services built from e6f8b80 (linux/amd64, Go 1.26, node-info. The branch node serves Branch node adds the 0.1.1 peer by address. 0.1.1 peer adds the branch node by address. Identical to before this PR: Take-away. The PR delivers once both ends run it; with a mixed fleet, the only visible change is the every-10-s rejection line on the older peer's proxies, which the description already notes for the node-info-down case and which applies equally to a not-yet-upgraded peer. A line in Could not exercise the both-sides path here, since the other nodes run release builds. Happy to re-run when a release carries this; the pod setup for it is in #31. |
cguldogan
added a commit
to cguldogan/Personal-AI-Router
that referenced
this pull request
Sep 8, 2026
feat/vllm-tailscale is the fork's combined branch: vLLM (PR NVIDIA#9), nodes across overlay networks such as Tailscale (PR NVIDIA#10), scripted headless pairing (PR NVIDIA#11), the fork's installer and release workflow, and now SGLang as a fourth engine. It is what the headless installer and the release packages build from.
7 tasks
ckelseynv
self-requested a review
September 10, 2026 02:29
Collaborator
|
Hi @cguldogan, thanks so much for your contributions! We are currently working on a Tailscale implementation from our roadmap and I'm looking into your PR right now. There are many things I'd love to bring in as is but may need to pull some out for a follow up PR. There is a bunch of work on our end regarding auto discovery, install, auth, etc that I plan on putting on top of your branch if that works for you? |
Author
Of course. However you’d like to proceed, and whatever is easiest for you to manage, works for me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PAIR finds peers by multicast DNS, which never leaves a local link. Two machines joined only by an overlay network (a Tailscale tailnet, a WireGuard tunnel, a routed link) were never going to discover each other, and adding such a peer by address did not work either: PAIR probed the peer's Ollama and LM Studio ports in plaintext, and on a machine running PAIR those ports are proxy front doors that refuse plaintext from anything but their own loopback. Every probe was refused, so a healthy peer reported no engines and nothing could be routed to it.
This makes a PAIR node added by address behave exactly like one found on a LAN: same card, engines, models, and telemetry, with inference routed to it over the same certificate-pinned mutual TLS.
nvpair-node-inforeports this node's service map on/v1/node-info, fed by the broker.nvpair-manual-nodesasks node-info first; a node that answers with a service map is a PAIR node and is never probed on its engine ports. Its inventory is read from its engine manager over cluster mTLS. A node without one is a bare inference host and keeps the original plaintext engine probes.host:portwith an actionable message, and gains a Service ports section for a node whose services are not on the defaultsnetpickadmits DNS names as dialable addresses. CGNAT (100.64/10) and overlay-interface demotion remain rankings, never exclusions, now pinned by tests.Scope
Included: the above, a cross-process test for a peer addehe desktop Add-node flow (create the node, then pair),
docs/remote-networks.mdxwith an ACL port list and troubleshooting, and README and SECURITY wording about the network boundary.Excluded: automatic discovery of tailnet peers (planned follow-up), and any code that knows about a specific VPN product. Nothing here is
Tailscale-specific; it makes PAIR work on any routed netw
Related: a sibling PR adds vLLM as a third engine. Both tal-nodes/manager.go
andservices/versions.json`;whichever merges second needs a small conflict resolution (vLLM becomes a third engine leg of the bare-host probe, and the higher version ofeach component is kept).
Validation
Environment: macOS 15 arm64, Go 1.27, Node 23; two Linux ver a Tailscale tailnet.
Pre-existing, environment-specific failures also present on
main:ollama-proxy TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone(needs a
127.0.0.2loopback alias), and the two tests t-port file from the real user data directory after anothertest writes it.Live check: two headless Linux nodes on a tailnet, each running PAIR from this branch, were added by address from a desktop node, paired with
the PIN exchange, and served inference from the desktop'sfore this change the same nodes appeared with no enginesand could not be paired.
Risk
Plaintext engine probes still run for bare hosts, as befo
node/addgains optionalports;/v1/node-infogainsservices; ahost:portaddress, previously accepted andpermanently unreachable, is now rejected. Additive otherw
node/remove-manualto the proxies on most probe cycles. It is cheap andidempotent, but it runs inline on the manual-nodes reader
peer logs a rejected plaintext request every 10 s until n
Versions: MINOR for
nvpair-node-info,nvpair-manual-noollama-proxy,lmstudio-proxy; PATCH fornvpair-node-scanner,nvpair-errors,nvpair-workload-manager,nvpair-cluster-manager`; product 0.91.7→0.92.0.Checklist
git commit -s), certificate of Origin](https://developercertificate.org/).Related: #9 (touches the same manual-nodes file; see the note under Scope) and #11.