Skip to content

Pair a headless machine from a script (nvpair-tui control socket and subcommands) - #11

Open
cguldogan wants to merge 6 commits into
NVIDIA:developfrom
cguldogan:pr/headless-pairing-cli
Open

cguldogan wants to merge 6 commits into
NVIDIA:developfrom
cguldogan:pr/headless-pairing-cli

Conversation

@cguldogan

@cguldogan cguldogan commented Sep 4, 2026

Copy link
Copy Markdown

Description

Accepting a pairing invitation on a headless machine meant pressing keys in the terminal interface's Cluster tab over SSH. This adds a scripted path.

A running nvpair-tui now serves a per-user JSON-RPC control socket, and the same binary invoked with a subcommand drives that instance:

nvpair-tui invite <address> [--port N] [--wait]   # prints the PIN
nvpair-tui pending
nvpair-tui accept --pin 123456 [--invite <id>] [--wait 2m]
nvpair-tui decline [--invite <id>]
nvpair-tui members

A desktop user can invite and the box can answer in one line, ssh gpu-box 'NVPAIR_PIN=402199 nvpair-tui accept', and two headless boxes can pair each other from a script. --json gives the raw result. Exit codes are a contract: 0 the pairing reached the state asked for, 1 the other side refused (wrong PIN, declined), 2 anything else.

The PIN stays a secret. It goes to the terminal the operator asked for and into the pair:invite result, nowhere else: pair:pending strips it, no error message repeats it, and nothing logs it (asss). NVPAIR_PIN is the documented scripted form, becausean argument is visible in ps and lands in shell history.

The interactive UI stays the source of truth. Pairing moved into one pairing.Service that both the Cluster tab and the socket call, so they
cannot disagree about which invitation is waiting: an invshows its PIN on the tab's status line exactly like i,and an accept from a script clears a PIN prompt the tab left open.

Scope

Included: the control socket ($XDG_RUNTIME_DIR/nvpair/tui.sock where the OS offers a runtime directory, else run/tui.sock under the shared
per-user data directory; a named pipe on Windows; directoale socket is reclaimed only after confirming nothinganswers on it; --control-socket overrides, --no-control-socket disables, and a socket that cannot be opened is a warning, not a failure),
the subcommands, unit tests, a cross-process test that paprocesses through real cluster managers on ephemeral ports, and docs in services/nvpair-tui/README.md and docs/terminal-interface.mdx.

Excluded: reconstructing pending invites after a TUI restart (the cluster manager exposes no list-invites method, and nodes:get-initial carries no invite id; documented), and any change to the the regenerated desktop/docs/services-api.md.

Independent of the two sibling PRs (vLLM engine, first-clies cleanly on main.

Validation

Environment: macOS 15 arm64, Go 1.27; Linux arm64 for the

cd services && ./build.sh                          # nvpair-tui 0.8.0
cd services/nvpair-tui && go test ./...            # 5 pa
cd services/tests && go test -run Headless ./...   # both cross-process tests pass
GOOS=windows go vet ./... ; GOOS=linux go vet ./... # cle
node scripts/spdx-headers.mjs                      # 0 missing
cd desktop && npm run service-contracts:write && npm run

The cross-process test runs each nvpair-tui under script(1) because Bubble Tea needs a TTY, and stands in a minimal stub broker for the
real one, since one machine can hold only one broker (itst is Unix-only.

Live: two headless Linux nodes running this build; `nvpaicket on each lists the cluster they were paired intoearlier. The invite/accept path itself was exercised by the cross-process test, not yet between the live nodes.

Risk

  • Security: the socket has no authentication beyond file mode / pipe DACL. Anything running as the same user can pair the machine, which is
    the authority the operator at the TUI already holds. The
  • Compatibility: additive. No broker or worker binary changes; nvpair-tui gains a socket and subcommands. invite --wait --json emits two
    JSON documents, one per line (the created invite with the), because a terminal invite carries no PIN.
  • Platform: Unix sun_path is capped at 104 bytes; an over-long path is rejected with the limit named rather than truncated, and
    --control-socket is the workaround. go-winio is addedindows named-pipe path.

Versions: nvpair-tui 0.7.2→0.8.0; product 0.91.7→0.92.0

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit mrivate data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results abov
  • I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

Related: #9 and #10. This PR is independent of both and applies cleanly on main.

cguldogan and others added 6 commits September 4, 2026 14:05
Accepting a pairing invitation on a headless GPU box meant pressing keys
in the Cluster tab. This adds the scripted path: a running nvpair-tui
serves a per-user JSON-RPC control socket, and `nvpair-tui invite`,
`pending`, `accept`, `decline` and `members` drive that instance from a
script or a single SSH command.

The pairing itself moves into nvpair-tui/pairing, one Service that both
drivers call, so the Cluster tab and the socket cannot hold different
ideas of which invite is pending: an invite created from a script shows
its PIN on the tab's status line exactly like `i`, and an accept made
there clears a PIN prompt the tab left open. Broker pushes are fanned
out in main so the service sees an invite whether or not the UI is
keeping up.

The endpoint is $XDG_RUNTIME_DIR/nvpair/tui.sock when the OS offers a
runtime directory and run/tui.sock under the shared per-user data
directory otherwise, its directory 0700 and the socket 0600; a socket
left by an instance that did not exit cleanly is reclaimed, but only
after confirming nothing answers on it. --control-socket overrides the
path and --no-control-socket disables it. Failing to open it warns and
leaves the UI running, because the UI is the primary job.

The PIN reaches the terminal the operator asked for and nowhere else:
pair:pending strips it, and NVPAIR_PIN keeps it out of shell history and
ps for scripts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The cross-process gate pairs two real nvpair-tui processes with no
keyboard: `invite` on one prints a PIN, `accept` on the other returns
paired, and `members` on both lists the pair. The pairing is the real
PIN-authenticated EAP-NOOB exchange between two real cluster managers.

Two substitutions, both because every broker-owned port is a compiled-in
constant and one machine can therefore hold only one broker, while this
needs two nodes: each TUI's broker is a stubbroker fixture that spawns a
real cluster manager on an ephemeral port and relays cluster:* / nodes:*
verbatim, and each TUI runs under script(1), because a Bubble Tea
program will not start without a terminal. Unix only, for the same
reason its process-group cleanup is.

Also fixed along the way: nvpair-tui's first import of nvpair-shared/ipc
needs go-winio in its own go.mod, or the Windows build cannot resolve
it; and `invite --wait --json` printed prose among the JSON. It now
prints two documents, one per line, the created invite carrying the PIN
and then its final state, because a terminal invite carries no PIN and
one document cannot serve both.

nvpair-tui 0.7.3 -> 0.8.0 (new IPC surface, additive), product and
installer 0.92.0 -> 0.93.0. No other binary's compiled output changed:
nothing under shared/ was touched, only imported.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
`npm run service-contracts:check` reads the `case "pair:…"` arms in
nvpair-tui's control server as methods that binary now handles, so
desktop/docs/services-api.md was stale. Regenerated with
service-contracts:write — the only desktop file this change touches, and
one that is generated, never hand-edited. The five methods list as "not
called by the bridge", which is correct: they are served on nvpair-tui's
own socket for its own subcommands, and the desktop application has no
business calling them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The two-headless-boxes example interpolated the PIN into the remote
command line, which is exactly what the paragraph above it says not to
do: it lands in the argument list of the local ssh and of the remote
shell, readable by anyone with an account on either machine for as long
as the pairing takes. It now travels over the connection's standard
input instead, and the difference is spelled out.

The one-off `ssh gpu-box 'NVPAIR_PIN=... nvpair-tui accept'` stays as it
is, and now says it is a one-off: someone has just read the digits off
another machine's screen, and the pairing is over in seconds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
The README said only a default path is length-checked. Listen checks
every path, including one given with --control-socket, which is the
case that actually bites: a long --control-socket path binds to a
truncated name nobody can dial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
`ping` reported brokerReady:true forever once app:ready had arrived, so
a script asking a TUI whose broker had died was told to carry on, and
its next invite could only time out. The notification fan-out now clears
the flag as it exits, which is the moment the broker's stream closes and
this process learns the broker is gone.

Two tests cover the fan-out's contract: that the pairing service sees an
invitation even when nothing is reading the UI's channel, and that
readiness is withdrawn on disconnect.

The stubbroker fixture gains a note that it deliberately does not
synthesize a reply for a relayed request a dead cluster manager never
answers — so nobody debugs a 35 s subcommand in that test twice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Can GULDOGAN <cguldogan@gmail.com>
@DustinTrap

Copy link
Copy Markdown

Validation from a Kubernetes pod, against a 0.1.1 inviter. Everything behaved as the description says; one exit-code observation at the end.

Environment. nvpair-tui 0.8.0 built from 5ca2475 (linux/amd64, Go 1.26, CGO_ENABLED=0), every other service the 0.1.1 release. It runs as a hostNetwork pod on single-node OpenShift 4.22 under an arbitrary UID, with XDG_RUNTIME_DIR on an emptyDir. The far side is a 0.1.1 headless Linux node (nvpair-tui 0.7.2 in tmux) that is already a member of a two-node cluster with a macOS desktop. The pod started with a fresh identity, so every pairing below is a real one.

Socket. Created at $XDG_RUNTIME_DIR/nvpair/tui.sock, mode 0600, owned by the arbitrary UID; control socket listening logged at startup. members --json and pending --json on the fresh node: exit 0, members: [], invites: [].

Accept (the headline case). The 0.1.1 node invited the pod by address from its Cluster tab; on the pod, NVPAIR_PIN=… nvpair-tui accept --wait 2m --json returned "state":"paired" with exit 0 about three seconds after the invite was sent. members then listed the whole cluster. pending --json before the accept carried no pin field, and the accept result has "pin":null.

Wrong PIN. NVPAIR_PIN=000000 … accept --wait 30s: "state":"failed","reason":"incorrect-pin", exit 1. The invite is consumed by the failed attempt (pending is empty afterwards), so the inviter has to invite again. Reasonable, maybe worth a sentence in the README.

Decline. Fresh invite, nvpair-tui decline --json: "state":"declined", exit 0.

Leave. L on the pod's Cluster tab removed it from the 0.1.1 peer's member list within a second or two. Did the join/leave cycle twice; no stale members either side.

Invite from the pod to a node already in a cluster. invite <addr> --wait --json returned {"reason":"already-clustered","state":"rejected"} with exit 2. That is the target's cluster manager refusing, so a script may expect it under exit 1 ("the other side refused") rather than 2 ("anything else"). Either is defensible; the README's exit-code contract could name which bucket a rejection lands in.

One thing noticed on the 0.1.1 side, not this PR: the inviter's status line stays at invite sent - PIN … after the invitee fails the PIN or declines. Just what a mixed-version operator will see.

For #31 this replaces the kubectl attach step with kubectl exec pair-0 -- nvpair-tui accept; I will drop that section once this merges.

cguldogan added a commit to cguldogan/Personal-AI-Router that referenced this pull request Sep 8, 2026
feat/vllm-tailscale is the fork's combined branch: vLLM (PR NVIDIA#9), nodes
across overlay networks such as Tailscale (PR NVIDIA#10), scripted headless
pairing (PR NVIDIA#11), the fork's installer and release workflow, and now
SGLang as a fourth engine. It is what the headless installer and the
release packages build from.
@sherief-nv
sherief-nv changed the base branch from main to develop September 11, 2026 17:11
@sherief-nv sherief-nv self-assigned this Sep 11, 2026
@sherief-nv

Copy link
Copy Markdown
Collaborator

Good idea, moved MR against develop.

@sherief-nv
sherief-nv requested review from Noah-Tervalon-Nvidia and sherief-nv and removed request for Noah-Tervalon-Nvidia September 11, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants