Make a node added by address a first-class peer (overlay networks such as Tailscale)

README.md

@@ -16,8 +16,12 @@ requests can be routed to eligible nodes according to engine availability,
 model availability, and current workload.
 
 PAIR is useful for concurrent local workloads such as multi-agent applications.
-Prompts and responses are intended to remain on the local network when every
-configured client, model source, engine, and node is local.
+Prompts and responses are intended to stay on the network you route them over —
+your local network, or an encrypted overlay such as a Tailscale tailnet that you
+configured yourself — when every configured client, model source, engine, and
+node is one of yours. Refer to
+[Running PAIR across a Tailscale tailnet](docs/remote-networks.mdx) for nodes
+that are not on the same local link.
 
 > PAIR routes each independent request to one node. It does **not** pool GPU
 > memory, combine GPUs into a larger logical GPU, shard one model across
@@ -224,19 +228,22 @@ Each entry assumes the ones before it.
 4. **[Terminal interface](docs/terminal-interface.mdx)** — the same tasks from a
    terminal, for a machine with no desktop environment. Skip it if every machine
    you run has a desktop.
-5. **[Troubleshooting](docs/troubleshooting.mdx)** — worth skimming once before
+5. **[Remote networks](docs/remote-networks.mdx)** — read this if your machines
+   are joined by a VPN or overlay network rather than a local link: discovery
+   does not cross one, so you add the peer by name instead.
+6. **[Troubleshooting](docs/troubleshooting.mdx)** — worth skimming once before
    you need it, so you know where the diagnostics live. Alongside it,
    **[Known issues](docs/known-issues.mdx)** lists the significant limitations we
    are already aware of, and
    **[Collecting and sanitizing logs](docs/log-collection.mdx)** covers preparing
    a log you can share.
-6. **[Architecture](docs/architecture.mdx)** — the process model, how a request is
+7. **[Architecture](docs/architecture.mdx)** — the process model, how a request is
    routed, and where the trust boundaries are. Read this before changing
    anything, or if you want to know why PAIR behaves the way it does.
-7. **[Building and running](docs/building.mdx)** — prerequisites, building from
+8. **[Building and running](docs/building.mdx)** — prerequisites, building from
    source, running the services without the desktop application, and writing
    your own client against the JSON-RPC API.
-8. **[Developer guide](docs/developing.mdx)** — read this before contributing:
+9. **[Developer guide](docs/developing.mdx)** — read this before contributing:
    where the code lives, how a change travels through the layers, and the
    conventions the project enforces.
 

SECURITY.md

@@ -48,7 +48,10 @@ PAIR is a LAN-first, multi-process application:
   JSON-RPC 2.0 over the broker's standard input and output.
 - The broker supervises Go workers and relays their control-plane methods and
   notifications.
-- Discovery and selected metadata endpoints operate on the local network.
+- Discovery and selected metadata endpoints operate on the local network, or on
+  an encrypted overlay network the operator configured, such as a Tailscale
+  tailnet. Discovery itself is multicast and does not cross one; a node on the
+  far side is added by address.
 - Ollama-compatible and OpenAI-compatible local HTTP proxies carry inference
   traffic and may route a request to another paired node.
 - The cluster manager uses a six-digit PIN to bootstrap trust. Cluster-scoped
@@ -76,17 +79,26 @@ open relay for inference to anything that can route to it. Run an application on
 a node and use that node's local endpoint. Exposing an engine to the network
 directly is outside PAIR and is the operator's decision and risk.
 
-### Local Network Is a Trust-Relevant Boundary
+### The Network You Route Over Is a Trust-Relevant Boundary
 
 PAIR discovers nodes and exposes service metadata on the LAN. Some discovery
 enrichment and node-information traffic can use plain HTTP. Treat an untrusted
 Wi-Fi, shared office network, compromised router, and hostile local process as
 potentially adversarial. Network segmentation and host firewall rules remain
 the operator's responsibility.
 
+Nodes joined by an encrypted overlay network such as a Tailscale tailnet, rather
+than by a local link, are the same boundary reached a different way. The overlay
+carries its own encryption and its own admission policy, and PAIR's inter-node
+traffic is certificate-pinned mutual TLS in either case; what changes is that
+everyone admitted to the overlay is on this boundary, so who may join it is the
+operator's decision. Refer to
+[Running PAIR across a Tailscale tailnet](docs/remote-networks.mdx).
+
 “Local-first” describes the intended topology. It does not prove that no data
-leaves the machine or LAN. Inference engines, model catalogs, update systems,
-applications, and user configuration may contact external services.
+leaves the machine, the LAN, or an overlay you configured. Inference engines,
+model catalogs, update systems, applications, and user configuration may contact
+external services.
 
 ### Pairing PIN Is a Bootstrap Convenience
 

desktop/src/electron/service-bridge/empty-handlers.ts

@@ -11,6 +11,7 @@ import {
     MODULAR_ENGINE_LIFECYCLE_CALL_TIMEOUT_MS
 } from '@/shared/constants/modular-runtime'
 import getErrorString from '@/shared/utils/get-error-string'
+import { createStructuredLogger } from '@/shared/utils/log'
 import { getEngineHubModels } from '@/electron/model-hub'
 import { getModularSupervisor } from './modular-supervisor'
 import {
@@ -23,7 +24,12 @@ import {
 import type { ProxyEngine } from './modular-state'
 import type { JsonObject, JsonValue } from './json-rpc-subprocess'
 import { emptyInvite, parseClusterNodes, parseInvite, parseNodeIdentity } from './cluster-json'
-import { removeManualNodeEntry, resolveManualNodeKey } from './manual-nodes-store'
+import {
+    addManualNodeEntry,
+    manualPortsToWire,
+    removeManualNodeEntry,
+    resolveManualNodeKey
+} from './manual-nodes-store'
 
 type BridgeHandler<C extends WsInvokeChannel> = (
     payload?: WsInvokeRequest<C>
@@ -34,6 +40,8 @@ type BridgeHandlerMap = {
 }
 
 // The broker allows pairing exchanges up to 30 seconds; keep the UI bridge outside that deadline.
+const log = createStructuredLogger('service-bridge')
+
 const CLUSTER_PAIRING_CALL_TIMEOUT_MS = 35_000
 
 function wait(ms: number): Promise<void> {
@@ -798,6 +806,32 @@ async function handleClusterInviteNode(
 ): Promise<Invite> {
     if (!payload) return emptyInvite()
 
+    // Add the address as a manual node BEFORE inviting it.
+    //
+    // On a network that carries no multicast — a Tailscale tailnet, a routed
+    // subnet — nothing is ever discovered, so a peer that is only paired stays
+    // invisible: no record arrives for it, ever. Adding it first also gives the
+    // operator the one piece of feedback that matters when an address was typed
+    // by hand: the node appears with its hardware as soon as it answers, whether
+    // or not the pairing that follows succeeds.
+    const supervisor = getModularSupervisor()
+    if (supervisor.hasProcess('broker')) {
+        const entry = addManualNodeEntry(payload.ipAddress, payload.ports)
+        const params: JsonObject = { address: entry.address, name: entry.name }
+        const wirePorts = manualPortsToWire(entry.ports)
+        if (wirePorts) params.ports = wirePorts
+        try {
+            await supervisor.callProcess('broker', 'node/add', params)
+        } catch (err) {
+            // Non-fatal: the entry is persisted and replayed on the next start,
+            // and the invite below is what the operator asked for.
+            log.warn({
+                sublevel: 'manual-nodes',
+                message: `Failed to add manual node ${entry.address}: ${getErrorString(err)}`
+            })
+        }
+    }
+
     // cluster-manager auto-founds a solo cluster on the first invite while
     // unclustered (under inviteMu, with invite-created provenance). Do not
     // pre-call cluster:create here: parallel Invites used to race concurrent
@@ -813,7 +847,7 @@ async function handleClusterInviteNode(
             'cluster:invite-node',
             {
                 address: payload.ipAddress,
-                port: MODULAR_CLUSTER_MANAGER_PORT
+                port: payload.ports?.cluster ?? MODULAR_CLUSTER_MANAGER_PORT
             },
             CLUSTER_PAIRING_CALL_TIMEOUT_MS
         )

desktop/src/electron/service-bridge/manual-nodes-store.ts

@@ -4,12 +4,19 @@
 import fs from 'fs'
 import path from 'path'
 import { getPaths } from '@/electron/globals'
+import type { ManualServicePorts } from '@/shared/types/manual-node'
 import type { JsonObject, JsonValue } from './json-rpc-subprocess'
 
 interface ManualNodeEntry {
     id: string
     address: string
     name: string
+    /**
+     * Per-service port overrides, persisted because the replay after a restart
+     * is the only thing that re-creates the entry: without them a node reachable
+     * on non-default ports comes back unreachable.
+     */
+    ports?: ManualServicePorts
 }
 
 function configFilePath(): string {
@@ -25,6 +32,57 @@ function stringValue(value: JsonValue | undefined): string {
     return typeof value === 'string' ? value : ''
 }
 
+/** A usable TCP port, or undefined for anything else. Narrowing, never a cast. */
+function portValue(value: JsonValue | undefined): number | undefined {
+    if (typeof value !== 'number' || !Number.isInteger(value)) return undefined
+    if (value < 1 || value > 65535) return undefined
+    return value
+}
+
+function portsValue(value: JsonValue | undefined): ManualServicePorts | undefined {
+    const obj = objectValue(value)
+    if (!obj) return undefined
+    const ports: ManualServicePorts = {
+        nodeInfo: portValue(obj.nodeInfo),
+        cluster: portValue(obj.cluster),
+        ollama: portValue(obj.ollama),
+        lmstudio: portValue(obj.lmstudio),
+        vllm: portValue(obj.vllm)
+    }
+    return definedPorts(ports)
+}
+
+/**
+ * Drops every unset field, and the object itself when nothing is set. An empty
+ * overrides object is not the same thing as none: it would persist and replay as
+ * a value the operator never chose.
+ */
+function definedPorts(ports: ManualServicePorts): ManualServicePorts | undefined {
+    const kept: ManualServicePorts = {}
+    if (ports.nodeInfo !== undefined) kept.nodeInfo = ports.nodeInfo
+    if (ports.cluster !== undefined) kept.cluster = ports.cluster
+    if (ports.ollama !== undefined) kept.ollama = ports.ollama
+    if (ports.lmstudio !== undefined) kept.lmstudio = ports.lmstudio
+    if (ports.vllm !== undefined) kept.vllm = ports.vllm
+    return Object.keys(kept).length > 0 ? kept : undefined
+}
+
+/**
+ * Projects the overrides onto the snake_case field names `node/add` reads
+ * (`node_info`, `cluster`, `ollama`, `lmstudio`, `vllm`). The two spellings meet
+ * here and nowhere else.
+ */
+export function manualPortsToWire(ports: ManualServicePorts | undefined): JsonObject | undefined {
+    if (!ports) return undefined
+    const wire: JsonObject = {}
+    if (ports.nodeInfo !== undefined) wire.node_info = ports.nodeInfo
+    if (ports.cluster !== undefined) wire.cluster = ports.cluster
+    if (ports.ollama !== undefined) wire.ollama = ports.ollama
+    if (ports.lmstudio !== undefined) wire.lmstudio = ports.lmstudio
+    if (ports.vllm !== undefined) wire.vllm = ports.vllm
+    return Object.keys(wire).length > 0 ? wire : undefined
+}
+
 function entryValue(value: JsonValue | undefined): ManualNodeEntry | null {
     const obj = objectValue(value)
     if (!obj) return null
@@ -34,7 +92,30 @@ function entryValue(value: JsonValue | undefined): ManualNodeEntry | null {
 
     const name = stringValue(obj.name) || address
     const id = stringValue(obj.id) || name
-    return { id, address, name }
+    const ports = portsValue(obj.ports)
+    return ports ? { id, address, name, ports } : { id, address, name }
+}
+
+/**
+ * Record a manually added node so it survives a restart, and hand back the entry
+ * the broker should be told about.
+ *
+ * `nvpair-manual-nodes` keeps no durable state of its own — the list belongs to
+ * the application — and the backend keys the node by `name`, which is the
+ * address here so {@link resolveManualNodeKey} and the `node/remove` relay agree
+ * on it. Re-adding the same address replaces its entry rather than duplicating
+ * it, so changing a node's ports is just adding it again.
+ */
+export function addManualNodeEntry(address: string, ports?: ManualServicePorts): ManualNodeEntry {
+    const trimmed = address.trim()
+    const kept = ports ? definedPorts(ports) : undefined
+    const entry: ManualNodeEntry = kept
+        ? { id: trimmed, address: trimmed, name: trimmed, ports: kept }
+        : { id: trimmed, address: trimmed, name: trimmed }
+    const entries = listManualNodeEntries().filter(existing => existing.address !== trimmed)
+    entries.push(entry)
+    saveManualNodeEntries(entries)
+    return entry
 }
 
 export function listManualNodeEntries(): ManualNodeEntry[] {

desktop/src/electron/service-bridge/modular-supervisor.ts

@@ -38,7 +38,7 @@ import {
     isModularLogLevel,
     type ModularLogLevel
 } from '@/shared/constants/modular-runtime'
-import { listManualNodeEntries } from './manual-nodes-store'
+import { listManualNodeEntries, manualPortsToWire } from './manual-nodes-store'
 import {
     MODULAR_RUNTIME_BINARIES,
     modularBinaryFileName
@@ -851,10 +851,10 @@ class ModularSupervisor {
         const entries = listManualNodeEntries()
         for (const entry of entries) {
             try {
-                await this.callProcess('broker', 'node/add', {
-                    address: entry.address,
-                    name: entry.name
-                })
+                const params: JsonObject = { address: entry.address, name: entry.name }
+                const wirePorts = manualPortsToWire(entry.ports)
+                if (wirePorts) params.ports = wirePorts
+                await this.callProcess('broker', 'node/add', params)
             } catch (err) {
                 log.warn({
                     sublevel: 'manual-nodes',

desktop/src/shared/types/manual-node.ts

@@ -0,0 +1,22 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+/**
+ * Per-service port overrides for a manually added node.
+ *
+ * The defaults `nvpair-manual-nodes` assumes describe a machine it cannot
+ * introspect: a peer may run an engine on a second port, a whole node may be
+ * reachable only through a forwarded range, and a test may put two nodes on one
+ * loopback. An unset field keeps that service's default, so an entry overrides
+ * only what the operator meant to.
+ *
+ * `vllm` is carried and persisted but not probed yet; it is here so an entry
+ * written today keeps its meaning when that engine lands.
+ */
+export interface ManualServicePorts {
+    nodeInfo?: number
+    cluster?: number
+    ollama?: number
+    lmstudio?: number
+    vllm?: number
+}

desktop/src/shared/types/ws-channels.ts

@@ -23,6 +23,7 @@
  *   - `export` is reserved for the handful of symbols consumed by the
  *     service bridge generics — nothing else.
  */
+import type { ManualServicePorts } from '@/shared/types/manual-node'
 import type {
     EngineCommandPayload,
     EngineHubSearchResponse,
@@ -75,7 +76,17 @@ export interface WsInvokeChannelMap {
 
     // Cluster — PIN-pairing handshake (nvpair-cluster-manager)
     'cluster:get-initial': { request: void; response: ClusterInitialSnapshot }
-    'cluster:invite-node': { request: { ipAddress: string }; response: Invite }
+    // Add a node by address and start pairing with it, in that order. The
+    // address is added as a manual node first so it appears with its hardware
+    // as soon as it answers -- on a network without multicast that is the only
+    // feedback the operator gets about whether the address is right, and it is
+    // also what keeps a successfully paired peer visible afterwards, since no
+    // discovery record will ever arrive for it. `ports` overrides the assumed
+    // port of any single service on that host.
+    'cluster:invite-node': {
+        request: { ipAddress: string; ports?: ManualServicePorts }
+        response: Invite
+    }
     'cluster:invite-status': { request: { inviteId: string }; response: Invite }
     'cluster:respond-to-invite': {
         request: { inviteId: string; accept: boolean; pin?: string }

desktop/src/ui/api/pair-api.ts

@@ -9,6 +9,7 @@ import type {
     ClusterNode,
     Invite
 } from '@/shared/types/cluster'
+import type { ManualServicePorts } from '@/shared/types/manual-node'
 import type { NodeItem } from '@/shared/types/nodes'
 import type { ServiceError } from '@/shared/types/errors'
 import type { NodeItemMetrics } from '@/shared/types/metrics'
@@ -51,7 +52,7 @@ export interface IClusterApi {
     /** Fetch cluster bootstrap state: identity, settings, and membership. */
     getInitial(): Promise<ClusterInitialSnapshot>
     /** Start PIN pairing with a remote node; the returned invite carries the PIN to display. */
-    inviteNode(ipAddress: string): Promise<Invite>
+    inviteNode(ipAddress: string, ports?: ManualServicePorts): Promise<Invite>
     /** Poll the state of an outbound pairing session. */
     inviteStatus(inviteId: string): Promise<Invite>
     /** Respond to an inbound invite: accept with the PIN from the inviter, or decline. */
@@ -152,7 +153,8 @@ export function createPairApi(transport: ServiceTransport): IPairApi {
         },
         cluster: {
             getInitial: () => transport.invoke('cluster:get-initial'),
-            inviteNode: ipAddress => transport.invoke('cluster:invite-node', { ipAddress }),
+            inviteNode: (ipAddress, ports) =>
+                transport.invoke('cluster:invite-node', { ipAddress, ports }),
             inviteStatus: inviteId => transport.invoke('cluster:invite-status', { inviteId }),
             respondToInvite: (inviteId, accept, pin) =>
                 transport.invoke('cluster:respond-to-invite', { inviteId, accept, pin }),