diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.md index 590e7264..e19201dd 100644 --- a/desktop/docs/services-api.md +++ b/desktop/docs/services-api.md @@ -32,6 +32,11 @@ - ⚠️ nvpair-node-settings → settings/get-force-ports - ⚠️ nvpair-node-settings → settings/set-cluster-auto-sync - ⚠️ nvpair-node-settings → settings/set-force-ports +- ⚠️ nvpair-tui → pair:invite +- ⚠️ nvpair-tui → pair:invite-status +- ⚠️ nvpair-tui → pair:members +- ⚠️ nvpair-tui → pair:pending +- ⚠️ nvpair-tui → pair:respond - ⚠️ nvpair-ui-broker → discovery:unsubscribe - ⚠️ nvpair-ui-broker → engine:set-reserved-port - ⚠️ nvpair-ui-broker → engine:unsubscribe @@ -208,12 +213,19 @@ | Method | Direction | In bridge? | |---|---|---| | `cluster:identity-changed` | request (we call) | ✅ yes | +| `cluster:invite-canceled` | request (we call) | ✅ yes | +| `cluster:invite-expired` | request (we call) | ✅ yes | | `cluster:invite-received` | request (we call) | ✅ yes | | `engine:install-progress` | request (we call) | ✅ yes | | `engine:pull-progress` | request (we call) | ✅ yes | | `engine:state-changed` | request (we call) | ✅ yes | | `error` | request (we call) | ✅ yes | | `nodes:changed` | request (we call) | ✅ yes | +| `pair:invite` | request (we call) | ⚠️ not called | +| `pair:invite-status` | request (we call) | ⚠️ not called | +| `pair:members` | request (we call) | ⚠️ not called | +| `pair:pending` | request (we call) | ⚠️ not called | +| `pair:respond` | request (we call) | ⚠️ not called | | `workloads:remove` | request (we call) | ✅ yes | | `workloads:upsert` | request (we call) | ✅ yes | diff --git a/docs/terminal-interface.mdx b/docs/terminal-interface.mdx index 1eb99b5f..3c52aa46 100644 --- a/docs/terminal-interface.mdx +++ b/docs/terminal-interface.mdx @@ -65,6 +65,8 @@ cd | --- | --- | | `--broker-path ` | Use a service binary that is not beside `nvpair-tui` | | `--log-level ` | Verbosity of the terminal interface's own logging: `debug`, `info`, `warn`, or `error`. PAIR also reads this from `NVPAIR_LOG_LEVEL` | +| `--control-socket ` | Serve the pairing control socket somewhere other than its default per-user path | +| `--no-control-socket` | Serve no control socket, which disables [pairing from a script](#pair-from-a-script-or-over-ssh) | | `--version` | Print the version and exit | The interface's own log output goes to stderr, so it never corrupts the display. @@ -185,6 +187,126 @@ Only pair when you trust both machines and the network. The PIN is a short-lived bootstrap code, not a durable credential. Refer to the [security policy](../SECURITY.md). +## Pair from a Script or Over SSH + +Every pairing step above is also a command. While the terminal interface is +running, the same `nvpair-tui` binary invoked with a command talks to it, so you +can pair a machine from a script, or in one `ssh` line, without opening the +interface and pressing keys. + +**The interface has to be running.** These commands drive it; they do not start +anything. Leave it running in tmux, as in +[Keep It Running After You Disconnect](#keep-it-running-after-you-disconnect). +If it is not running, every command exits `2` and says so. They also do not +reach the desktop application — that is a different program with its own +services, and the two must never run on one machine anyway. + +```bash +nvpair-tui invite
[--port N] [--wait] +nvpair-tui pending +nvpair-tui accept --pin 123456 [--invite ] [--wait 2m] +nvpair-tui decline [--invite ] +nvpair-tui members +``` + +Add `--json` to any of them to get the raw result instead of a human line, for a +script to parse. + +**Put the PIN in `NVPAIR_PIN`, not in `--pin`, in a script.** `--pin 402199` is +visible to every user on the machine for as long as the command runs, because +process arguments are public in `ps`, and it is written to your shell history. +An environment variable is neither. + +```bash +NVPAIR_PIN=402199 nvpair-tui accept +``` + +`--pin` is for typing at an interactive prompt, where you have just read the +digits off another machine's screen and the pairing is over in seconds. + +### The Desktop Invites, the Headless Box Accepts + +This is the common case: someone at the desktop application starts the pairing, +and the machine that has to answer has no screen. + +1. On the desktop machine, invite the box by address and note the six digits it + shows. +2. On the box, over SSH: + + ```bash + ssh gpu-box 'NVPAIR_PIN=402199 nvpair-tui accept' + ``` + + It exits `0` once the machines are paired. Typing the digits like this is + fine for a one-off; see [Two Headless Boxes](#two-headless-boxes) for the + form to use in a script. + +If you want to arm the accept before the desktop user has sent anything, give it +a window to wait in: + +```bash +NVPAIR_PIN=402199 nvpair-tui accept --wait 2m +``` + +It waits up to two minutes for an invitation to arrive, then answers it. + +### The Headless Box Invites, the Desktop Accepts + +Run the invitation on the box and read out the PIN it prints: + +```bash +$ nvpair-tui invite 192.168.1.40 +PIN 402199 invite inv-9f3a1c to 192.168.1.40 +Read the PIN to whoever is at that machine; they run: nvpair-tui accept --pin 402199 +``` + +The desktop user types those six digits into the prompt the desktop application +shows. Add `--wait` and the command blocks until the pairing finishes, exiting +`0` when it succeeds. + +### Two Headless Boxes + +Neither machine has a screen, so the PIN goes from one command's output straight +into the other's environment, over the SSH connection's standard input rather +than through either machine's command line: + +```bash +PIN=$(ssh box-a 'nvpair-tui invite box-b.example.net --json' | jq -r .pin) +echo "$PIN" | ssh box-b 'read -r NVPAIR_PIN; export NVPAIR_PIN; nvpair-tui accept --wait 2m' +ssh box-a 'nvpair-tui members' +``` + +The last line lists the cluster and both machines, which is the confirmation +that it worked. + +The `read` is what keeps the PIN out of `ps` on both machines. Interpolating it +into the remote command — `ssh box-b "NVPAIR_PIN=$PIN ..."` — is easier to read +but puts the digits in the argument list of the local `ssh` and of the remote +shell, where anyone with an account on either machine can see them for as long +as the pairing takes. + +### Exit Codes + +| Code | Meaning | +| --- | --- | +| `0` | The pairing reached the state you asked for | +| `1` | The other machine refused: a wrong PIN, or a declined invitation | +| `2` | Anything else: no terminal interface running, a bad argument, an unreachable machine, an expired invitation | + +### What You Can and Cannot Do This Way + +`pending` lists only invitations that arrived while this terminal interface has +been running. If you restart it while an invitation is in flight, `pending` +comes back empty even though the other machine still thinks one is open. Send a +new invitation. + +`--invite ` is only needed when more than one invitation is waiting at once. +With exactly one, the commands find it; with none or several, they say so and +name them. + +Pairing is all these commands do. Everything else — engines, models, proxies, +settings — is still the interface's tabs. + ## Prepare an Engine and a Model From the **Engines** tab (6), select an engine with `j` / `k`, then: diff --git a/services/bom.md b/services/bom.md index 63497a36..917704b4 100644 --- a/services/bom.md +++ b/services/bom.md @@ -21,7 +21,7 @@ As of the mDNS dedup, `grandcat/zeroconf`, `miekg/dns`, and `golang.org/x/net` a | Library | Version | Used By | License | License URL | |---------|---------|---------|---------|-------------| -| `github.com/Microsoft/go-winio` | v0.6.2 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) | +| `github.com/Microsoft/go-winio` | v0.6.2 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler, nvpair-tui | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) | | `github.com/charmbracelet/bubbles` | v1.0.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbles/blob/master/LICENSE) | | `github.com/charmbracelet/bubbletea` | v1.3.10 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbletea/blob/master/LICENSE) | | `github.com/charmbracelet/lipgloss` | v1.1.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/lipgloss/blob/master/LICENSE) | diff --git a/services/nvpair-tui/README.md b/services/nvpair-tui/README.md index e5b92ca7..743e91c8 100644 --- a/services/nvpair-tui/README.md +++ b/services/nvpair-tui/README.md @@ -13,6 +13,10 @@ for the graphical UI, and it does not cover every operation the desktop does. It spawns and owns its own `nvpair-ui-broker` child over stdio; the broker in turn supervises the worker subprocesses, so `nvpair-tui` drives one host on its own. +While it runs it also serves a **local control socket**, so pairing can be +driven from a script instead of a keyboard — see +[Pairing without a keyboard](#pairing-without-a-keyboard). + This file is the component reference. For task-oriented usage instructions, see [Using the PAIR terminal interface](../../docs/terminal-interface.mdx). @@ -56,18 +60,129 @@ installed `bin/` layout). Override with `--broker-path`: nvpair-tui # broker is a sibling binary nvpair-tui --broker-path /opt/nvpair/bin/nvpair-ui-broker nvpair-tui --log-level debug # own logging (to stderr) +nvpair-tui --control-socket /run/pair.sock # serve the endpoint somewhere else +nvpair-tui --no-control-socket # serve no endpoint at all nvpair-tui --version ``` Logging goes to stderr (the broker's logs are shown inside the **Logs** tab, not on the terminal), so it never corrupts the full-screen UI. +## Pairing without a keyboard + +A running `nvpair-tui` listens on a per-user JSON-RPC control socket, and the +same binary invoked with a subcommand connects to it. That is how an operator +pairs a headless box over SSH, or from a script, without pressing keys in the +Cluster tab. + +The subcommands reach a running **`nvpair-tui`**, not the desktop application's +broker. On a machine where only the desktop application is running, they report +that nothing is listening — which is correct: the two must never run at once. + +### Subcommands + +```sh +nvpair-tui invite
[--port N] [--wait] # prints the PIN on stdout +nvpair-tui pending # inbound invitations +nvpair-tui accept --pin 123456 [--invite ] # answer one +nvpair-tui accept --pin 123456 --wait 2m # wait for one, then answer it +nvpair-tui decline [--invite ] +nvpair-tui members # cluster id and members +``` + +Every subcommand takes `--json`, which prints the endpoint's raw result instead +of a human line, and `--control-socket ` to reach an instance started +with a non-default endpoint. `invite --wait --json` prints two JSON objects, +one per line: the invite as created, which carries the PIN, and then its final +state (an invite in a terminal state carries no PIN, so one document cannot +serve both). + +`--invite` is only needed when more than one invitation is waiting; with +exactly one, the endpoint resolves it, and with none or several it says so. + +**Pass the PIN in `NVPAIR_PIN`, not `--pin`, in a script.** An argument is +visible to every process on the machine in `ps` and is written to shell +history; the environment form is neither. `--pin` exists for typing at an +interactive prompt. + +### Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | The pairing reached the state asked for: `paired` for `accept` and `invite --wait`, `declined` for `decline`. | +| `1` | The other side refused: an incorrect PIN, or a declined invitation. | +| `2` | Anything else: no running `nvpair-tui`, a bad argument, an unreachable peer, an expired or rejected invite, a transport failure. | + +### The control socket + +| | | +| --- | --- | +| **Path** | `$XDG_RUNTIME_DIR/nvpair/tui.sock` when the OS provides a runtime directory; otherwise `run/tui.sock` under the shared per-user data directory (`nvpair-shared/appdir`). On Windows, the named pipe `\\.\pipe\nvpair-tui-`. | +| **Protocol** | Newline-delimited JSON-RPC 2.0, the same as every other PAIR surface. Request/response only; the endpoint pushes nothing. | +| **Access** | The directory is `0700` and the socket `0600`, so the file permissions are the check. On Windows the pipe's default DACL grants the creating user. Anything that can open it can pair this machine, which is the authority the operator at the TUI already has. | +| **Clients** | Any number, served concurrently, each issuing any number of sequential requests. | +| **Overrides** | `--control-socket `, or `--no-control-socket` to serve none. | + +A socket file left behind by an instance that did not exit cleanly is removed +and reclaimed, but only after confirming nothing answers on it — a live socket +means another `nvpair-tui` is running, and this one refuses to steal it. If the +endpoint cannot be opened at all, `nvpair-tui` logs a warning and runs the +interactive UI anyway: the UI is its primary job. + +On Unix the path has to fit the platform's `sun_path` limit (104 bytes on +macOS). Any path that would not fit — the default or one given with +`--control-socket` — is reported with that limit named, rather than being +silently truncated by the kernel into a socket nobody will dial. + +### Methods + +Each is relayed through the TUI's one broker connection, so the broker keeps a +single client. Where the answer is a cluster-manager `Invite`, the manager's own +JSON is relayed untouched. + +| Method | Params | Result | +| --- | --- | --- | +| `ping` | — | `{version, brokerReady}` — `brokerReady` goes back to `false` if the broker dies, so a script is not sent on to an invite that can only time out | +| `pair:invite` | `{address, port?, nodeId?}` | the `Invite`, including its `pin` | +| `pair:invite-status` | `{inviteId}` | the current `Invite` | +| `pair:pending` | — | `{invites: [...]}` — inbound invitations not yet answered, each with the `pin` member removed and a `receivedAt` (this node's clock, epoch ms) added so a caller can age it | +| `pair:respond` | `{inviteId?, accept, pin?}` | the resulting `Invite` | +| `pair:members` | — | `{clusterId, clusterFriendlyName, nodeId, nodeUuid, name, members: [ClusterNode]}` | + +`Invite` and `ClusterNode` are `nvpair-cluster-manager`'s own shapes; see its +[README](../nvpair-cluster-manager/README.md). + +Errors relay the cluster manager's code and message verbatim, so `-32001` +(unknown invite) and `-32002` (invalid invite state) mean there what they mean +in the manager. Two codes originate here: + +| Code | Meaning | +| --- | --- | +| `-32010` | `pair:respond` named no invite and none is waiting | +| `-32011` | `pair:respond` named no invite and several are waiting; `data.invites` lists them | + +### What it does not do + +**The pending set is session state.** `nvpair-cluster-manager` has no "list the +invitations you are holding" call, and `nodes:get-initial` reports a +`pending-inbound` peer without the invite id needed to answer it. So an +`nvpair-tui` restarted while an invitation was in flight reports nothing +pending even though the manager still holds a live invite; the inviting machine +has to send a new one. Nothing here reconstructs it. + +**The PIN never reaches a log.** It goes to the terminal the operator asked for +and into the `pair:invite` result, and nowhere else: `pair:pending` strips it, +and no error message repeats one back. + ## Architecture ``` nvpair-tui (this process) ├── supervisor.go spawn/own nvpair-ui-broker over stdio, graceful teardown +├── cli.go the subcommands: connect to a running instance's socket ├── rpc/ JSON-RPC 2.0 codec + id-matching client +├── pairing/ the one pairing implementation both drivers call +├── control/ the local control socket: path, listener, server, client └── ui/ Bubble Tea root model + one file per tab │ stdio (newline-delimited JSON-RPC 2.0) ▼ @@ -77,6 +192,14 @@ nvpair-tui (this process) The supervisor sends `shutdown` and closes the broker's stdin on exit; the broker tears its own workers down, so quitting leaves no orphans. +`pairing.Service` is deliberately the only place pairing happens. The Cluster +tab and the control socket both call it, so they cannot disagree about which +invitation is waiting: an invite created from a script shows its PIN on the +tab's status line exactly as one created with `i`, and an accept made from a +script clears a PIN prompt the tab left open. Broker notifications are fanned +out in `main.go` — to the service first, then the UI — so the service sees an +invitation arrive whether or not the UI is keeping up. + ## Build & test Built by the repo's top-level `build.bat` / `build.sh` (stamped via diff --git a/services/nvpair-tui/cli.go b/services/nvpair-tui/cli.go new file mode 100644 index 00000000..c25d2597 --- /dev/null +++ b/services/nvpair-tui/cli.go @@ -0,0 +1,668 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +// The non-interactive half of nvpair-tui. `nvpair-tui ` does not +// start a broker or a UI: it connects to the control socket of a TUI that is +// already running on this machine and drives that TUI's pairing, so an +// operator on a headless box can pair from a script or a single SSH command +// instead of pressing keys in the Cluster tab. +// +// Everything here is presentation and exit codes. The pairing itself lives in +// nvpair-tui/pairing, on the other side of the socket, which is the same code +// the Cluster tab drives. + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "sort" + "strings" + "time" + + "nvpair-tui/control" + "nvpair-tui/pairing" +) + +// Exit codes. They are a contract: a script keys on them, so they are +// documented in the component README and in docs/terminal-interface.mdx. +const ( + // exitOK — the pairing reached the state the caller asked for. + exitOK = 0 + // exitRefused — the other side said no: a wrong PIN, or a declined + // invite. The command worked; the answer was negative. + exitRefused = 1 + // exitFailure — anything else: no running TUI, a bad argument, an + // unreachable peer, an expired invite, a transport failure. + exitFailure = 2 +) + +// pinEnvVar is the scripted way to supply a PIN. A PIN passed as --pin lands +// in shell history and in every `ps` listing on the machine; the environment +// form keeps it out of both, and is the one the documentation recommends. +const pinEnvVar = "NVPAIR_PIN" + +// waitPoll is how often a --wait loop re-asks. nvpair-cluster-manager expires +// an unanswered invite after five minutes, so nothing here needs to be brisk. +const waitPoll = 2 * time.Second + +// inviteWaitCap bounds `invite --wait`. The invite's own TTL terminates the +// wait long before this; the cap only guarantees the command cannot hang +// forever if the manager stops answering. +const inviteWaitCap = 15 * time.Minute + +// dialTimeout bounds one control-socket request. It sits above the endpoint's +// own relay budget so a slow pairing surfaces as the manager's answer rather +// than as a timeout here. +const dialTimeout = 40 * time.Second + +// subcommands are the verbs that skip the UI entirely. +var subcommands = map[string]func(*cliEnv, []string) int{ + "invite": cmdInvite, + "pending": cmdPending, + "accept": cmdAccept, + "decline": cmdDecline, + "members": cmdMembers, +} + +// controlClient is the part of control.Client the subcommands use, so their +// argument handling, output and exit codes are testable without a socket. +type controlClient interface { + Call(ctx context.Context, method string, params any) (json.RawMessage, error) + Close() error +} + +// cliEnv is everything a subcommand touches outside itself. +type cliEnv struct { + out io.Writer + errOut io.Writer + getenv func(string) string + now func() time.Time + // poll is how long a --wait loop sleeps between asks. A test shortens it + // so the loop's logic can be exercised without the wall clock. + poll time.Duration + // dial opens a control connection. The default resolves the endpoint the + // running TUI listens on; a test substitutes its own. + dial func(path string) (controlClient, error) +} + +func newCLIEnv(out, errOut io.Writer) *cliEnv { + return &cliEnv{ + out: out, + errOut: errOut, + getenv: os.Getenv, + now: time.Now, + poll: waitPoll, + dial: func(path string) (controlClient, error) { + return control.Dial(path) + }, + } +} + +// subcommandName returns the verb in args, or "" when args carry none. A +// leading "-" is a flag, which means the caller wants the interactive UI. +func subcommandName(args []string) string { + if len(args) == 0 || strings.HasPrefix(args[0], "-") { + return "" + } + if _, ok := subcommands[args[0]]; ok { + return args[0] + } + return args[0] // an unknown verb, reported by run +} + +// runSubcommand executes the verb in args and returns the process exit code. +// It must only be called when subcommandName(args) is non-empty. +func runSubcommand(env *cliEnv, args []string) int { + name := args[0] + cmd, ok := subcommands[name] + if !ok { + fmt.Fprintf(env.errOut, "nvpair-tui: unknown command %q\n\n", name) + printUsage(env.errOut) + return exitFailure + } + return cmd(env, args[1:]) +} + +func printUsage(w io.Writer) { + fmt.Fprintf(w, `nvpair-tui runs the terminal interface. With a command, it instead drives the +pairing of an nvpair-tui already running on this machine, over its control socket. + + nvpair-tui start the terminal interface + nvpair-tui invite
[--port N] [--wait] + nvpair-tui pending + nvpair-tui accept --pin 123456 [--invite ] [--wait 2m] + nvpair-tui decline [--invite ] + nvpair-tui members + +Every command takes --json to print the raw result, and --control-socket +to reach a terminal interface started with a non-default endpoint. + +Read the PIN from %s instead of --pin in a script: an argument is visible to +every process on the machine and is kept in shell history. + +Exit codes: 0 the pairing reached the asked-for state; 1 the other side refused +(wrong PIN, or a declined invite); 2 anything else. +`, pinEnvVar) +} + +// commonFlags are the two flags every subcommand shares. +type commonFlags struct { + json *bool + socket *string +} + +func registerCommon(fs *flag.FlagSet) commonFlags { + return commonFlags{ + json: fs.Bool("json", false, "print the raw JSON result instead of a human-readable line"), + socket: fs.String("control-socket", "", "control socket of the running terminal interface (default: the per-user path)"), + } +} + +// parseArgs parses flags that may appear before, between or after positional +// arguments — Go's flag package stops at the first non-flag, so +// `invite 10.0.0.5 --port 14321` needs the parse resumed past each positional. +func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) { + var positional []string + rest := args + for { + if err := fs.Parse(rest); err != nil { + return nil, err + } + if fs.NArg() == 0 { + return positional, nil + } + positional = append(positional, fs.Arg(0)) + rest = fs.Args()[1:] + } +} + +// connect resolves the endpoint and opens a control connection to the running +// terminal interface. +func (env *cliEnv) connect(socket string) (controlClient, error) { + path := socket + if path == "" { + resolved, err := control.DefaultPath() + if err != nil { + return nil, err + } + path = resolved + } + return env.dial(path) +} + +// fail reports err and picks the exit code. A control endpoint nobody is +// listening on is the one failure worth explaining rather than just naming. +func (env *cliEnv) fail(err error) int { + if errors.Is(err, control.ErrNotRunning) { + fmt.Fprintf(env.errOut, "nvpair-tui: %v\n", err) + fmt.Fprintln(env.errOut, "Start the terminal interface on this machine first (nvpair-tui, ideally inside tmux), then run this command again.") + return exitFailure + } + fmt.Fprintf(env.errOut, "nvpair-tui: %v\n", err) + return exitFailure +} + +// callWithClient opens a connection, hands it to fn, and closes it. +func (env *cliEnv) callWithClient(socket string, fn func(context.Context, controlClient) int) int { + client, err := env.connect(socket) + if err != nil { + return env.fail(err) + } + defer func() { _ = client.Close() }() + return fn(context.Background(), client) +} + +// emit prints the raw result when --json was asked for, and otherwise runs the +// human renderer. It always returns code so a caller can `return env.emit(...)`. +func (env *cliEnv) emit(asJSON bool, raw json.RawMessage, human func(), code int) int { + if asJSON { + fmt.Fprintln(env.out, strings.TrimSpace(string(raw))) + return code + } + human() + return code +} + +// --- invite ----------------------------------------------------------------- + +func cmdInvite(env *cliEnv, args []string) int { + fs := flag.NewFlagSet("invite", flag.ContinueOnError) + fs.SetOutput(env.errOut) + common := registerCommon(fs) + port := fs.Int("port", 0, "pairing port on the target (default: the cluster manager's own, 14321)") + wait := fs.Bool("wait", false, "block until the pairing is accepted, refused, or expires") + positional, err := parseArgs(fs, args) + if err != nil { + return exitFailure + } + if len(positional) != 1 { + fmt.Fprintln(env.errOut, "nvpair-tui invite: exactly one address is required") + fmt.Fprintln(env.errOut, "usage: nvpair-tui invite
[--port N] [--wait] [--json]") + return exitFailure + } + address := positional[0] + + return env.callWithClient(*common.socket, func(ctx context.Context, client controlClient) int { + params := map[string]any{"address": address} + if *port != 0 { + params["port"] = *port + } + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:invite", params) + cancel() + if err != nil { + return env.fail(err) + } + var invite pairing.Invite + if err := json.Unmarshal(raw, &invite); err != nil { + return env.fail(fmt.Errorf("decode the invite: %w", err)) + } + + if invite.State == pairing.StateRejected { + return env.emit(*common.json, raw, func() { + fmt.Fprintf(env.errOut, "%s refused the invitation (%s). Remove the existing relationship on that machine first.\n", + address, reasonText(invite.Reason)) + }, exitFailure) + } + if invite.PIN() == "" { + return env.emit(*common.json, raw, func() { + fmt.Fprintf(env.errOut, "the invitation to %s did not start (state %s%s)\n", + address, invite.State, reasonSuffix(invite.Reason)) + }, exitFailure) + } + + if !*wait { + return env.emit(*common.json, raw, func() { + // The PIN goes to the terminal the operator asked for, and + // nowhere else. It is never logged. + fmt.Fprintf(env.out, "PIN %s invite %s to %s\n", invite.PIN(), invite.InviteID, address) + fmt.Fprintln(env.out, "Read the PIN to whoever is at that machine; they run: nvpair-tui accept --pin "+invite.PIN()) + }, exitOK) + } + + // Waiting still has to show the PIN straight away — it is what the + // operator reads out while the command blocks. Under --json that + // makes the output two JSON objects, one per line: the invite as + // created, carrying the PIN, and then its final state. The final + // invite carries no PIN, so a single document could not serve both. + env.emit(*common.json, raw, func() { + fmt.Fprintf(env.out, "PIN %s invite %s to %s\n", invite.PIN(), invite.InviteID, address) + fmt.Fprintln(env.out, "Waiting for the other machine to accept...") + }, exitOK) + final, err := env.awaitInvite(ctx, client, invite.InviteID) + if err != nil { + return env.fail(err) + } + return env.reportOutcome(*common.json, final, "the invitation") + }) +} + +// awaitInvite polls one invite until it leaves the pending state. +func (env *cliEnv) awaitInvite(ctx context.Context, client controlClient, inviteID string) (rawInvite, error) { + deadline := env.now().Add(inviteWaitCap) + for { + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:invite-status", map[string]any{"inviteId": inviteID}) + cancel() + if err != nil { + return rawInvite{}, err + } + var invite pairing.Invite + if err := json.Unmarshal(raw, &invite); err != nil { + return rawInvite{}, fmt.Errorf("decode the invite status: %w", err) + } + if invite.Terminal() { + return rawInvite{raw: raw, invite: invite}, nil + } + if !env.now().Before(deadline) { + return rawInvite{}, fmt.Errorf("invite %s was still pending after %s", inviteID, inviteWaitCap) + } + select { + case <-ctx.Done(): + return rawInvite{}, ctx.Err() + case <-time.After(env.poll): + } + } +} + +// rawInvite pairs the endpoint's verbatim answer with its decode. +type rawInvite struct { + raw json.RawMessage + invite pairing.Invite +} + +// reportOutcome renders a terminal invite and maps its state to an exit code. +// subject names what reached that state, for the human line. +func (env *cliEnv) reportOutcome(asJSON bool, res rawInvite, subject string) int { + inv := res.invite + code := exitFailure + switch { + case inv.State == pairing.StatePaired: + code = exitOK + case inv.State == pairing.StateDeclined: + code = exitRefused + case inv.State == pairing.StateFailed && inv.Reason == pairing.ReasonIncorrectPin: + code = exitRefused + } + return env.emit(asJSON, res.raw, func() { + line := fmt.Sprintf("%s: %s%s", subject, inv.State, reasonSuffix(inv.Reason)) + if code == exitOK { + fmt.Fprintln(env.out, line) + return + } + fmt.Fprintln(env.errOut, line) + }, code) +} + +// --- pending ---------------------------------------------------------------- + +func cmdPending(env *cliEnv, args []string) int { + fs := flag.NewFlagSet("pending", flag.ContinueOnError) + fs.SetOutput(env.errOut) + common := registerCommon(fs) + positional, err := parseArgs(fs, args) + if err != nil { + return exitFailure + } + if len(positional) != 0 { + fmt.Fprintln(env.errOut, "nvpair-tui pending: takes no arguments") + return exitFailure + } + + return env.callWithClient(*common.socket, func(ctx context.Context, client controlClient) int { + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:pending", nil) + cancel() + if err != nil { + return env.fail(err) + } + var result struct { + Invites []pairing.Invite `json:"invites"` + } + if err := json.Unmarshal(raw, &result); err != nil { + return env.fail(fmt.Errorf("decode the pending invites: %w", err)) + } + return env.emit(*common.json, raw, func() { + if len(result.Invites) == 0 { + fmt.Fprintln(env.out, "No invitations are waiting for an answer on this machine.") + return + } + // The invite itself carries no address, so the roster is asked + // for one. It is a convenience: a lookup that fails just leaves + // the address off the line. + addresses := env.inviterAddresses(ctx, client) + for _, inv := range result.Invites { + from := inv.FromNodeName + if from == "" { + from = inv.FromNodeID + } + if addr, ok := addresses[inv.FromNodeUUID]; ok && addr != "" { + from += " (" + addr + ")" + } + fmt.Fprintf(env.out, "%s from %s %s ago\n", inv.InviteID, from, env.since(inv)) + } + }, exitOK) + }) +} + +// inviterAddresses maps node uuid to the address the roster last saw it at, +// for the nodes whose pairing is still in flight. +func (env *cliEnv) inviterAddresses(ctx context.Context, client controlClient) map[string]string { + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + defer cancel() + raw, err := client.Call(callCtx, "pair:members", nil) + if err != nil { + return nil + } + var membership pairing.Membership + if err := json.Unmarshal(raw, &membership); err != nil { + return nil + } + out := make(map[string]string, len(membership.Members)) + for _, node := range membership.Members { + if node.NodeUUID != "" && node.IPAddress != "" { + out[node.NodeUUID] = node.IPAddress + } + } + return out +} + +// since renders how long an invite has been waiting, from this node's own +// clock. The inviter's createdAt is on a machine whose clock may differ, so it +// is only the fallback. +func (env *cliEnv) since(inv pairing.Invite) string { + stamp := inv.ReceivedAt + if stamp == 0 { + stamp = inv.CreatedAt + } + if stamp == 0 { + return "unknown" + } + age := env.now().Sub(time.UnixMilli(stamp)) + if age < 0 { + age = 0 + } + return age.Truncate(time.Second).String() +} + +// --- accept / decline ------------------------------------------------------- + +func cmdAccept(env *cliEnv, args []string) int { + fs := flag.NewFlagSet("accept", flag.ContinueOnError) + fs.SetOutput(env.errOut) + common := registerCommon(fs) + pin := fs.String("pin", "", "the six-digit PIN shown on the inviting machine (prefer "+pinEnvVar+" in a script)") + invite := fs.String("invite", "", "invite id to answer (default: the one that is pending)") + wait := fs.Duration("wait", 0, "if nothing is pending yet, wait this long for an invitation to arrive (e.g. 2m)") + positional, err := parseArgs(fs, args) + if err != nil { + return exitFailure + } + if len(positional) != 0 { + fmt.Fprintf(env.errOut, "nvpair-tui accept: unexpected argument %q\n", positional[0]) + return exitFailure + } + + resolvedPin := *pin + if resolvedPin == "" { + resolvedPin = strings.TrimSpace(env.getenv(pinEnvVar)) + } + if resolvedPin == "" { + fmt.Fprintf(env.errOut, "nvpair-tui accept: a PIN is required; pass --pin, or set %s (which keeps it out of shell history and ps)\n", pinEnvVar) + return exitFailure + } + + return env.callWithClient(*common.socket, func(ctx context.Context, client controlClient) int { + inviteID := *invite + if inviteID == "" && *wait > 0 { + resolved, code := env.awaitPending(ctx, client, *wait) + if code != exitOK { + return code + } + inviteID = resolved + } + return env.respond(ctx, client, *common.json, inviteID, true, resolvedPin) + }) +} + +func cmdDecline(env *cliEnv, args []string) int { + fs := flag.NewFlagSet("decline", flag.ContinueOnError) + fs.SetOutput(env.errOut) + common := registerCommon(fs) + invite := fs.String("invite", "", "invite id to decline (default: the one that is pending)") + positional, err := parseArgs(fs, args) + if err != nil { + return exitFailure + } + if len(positional) != 0 { + fmt.Fprintf(env.errOut, "nvpair-tui decline: unexpected argument %q\n", positional[0]) + return exitFailure + } + return env.callWithClient(*common.socket, func(ctx context.Context, client controlClient) int { + return env.respond(ctx, client, *common.json, *invite, false, "") + }) +} + +// respond answers one invite and maps the manager's verdict to an exit code. +func (env *cliEnv) respond(ctx context.Context, client controlClient, asJSON bool, inviteID string, accept bool, pin string) int { + params := map[string]any{"accept": accept} + if inviteID != "" { + params["inviteId"] = inviteID + } + if accept { + params["pin"] = pin + } + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:respond", params) + cancel() + if err != nil { + return env.fail(err) + } + var invite pairing.Invite + if err := json.Unmarshal(raw, &invite); err != nil { + return env.fail(fmt.Errorf("decode the response: %w", err)) + } + subject := "the invitation" + if !accept { + if invite.State == pairing.StateDeclined { + return env.emit(asJSON, raw, func() { + fmt.Fprintln(env.out, "declined the invitation") + }, exitOK) + } + return env.emit(asJSON, raw, func() { + fmt.Fprintf(env.errOut, "%s: %s%s\n", subject, invite.State, reasonSuffix(invite.Reason)) + }, exitFailure) + } + if invite.State == pairing.StatePaired { + return env.emit(asJSON, raw, func() { + fmt.Fprintf(env.out, "paired with %s\n", inviterName(invite)) + }, exitOK) + } + return env.reportOutcome(asJSON, rawInvite{raw: raw, invite: invite}, subject) +} + +// awaitPending blocks until an invitation is waiting, returning its id. It is +// what `accept --wait ` uses so an operator can arm the accept +// before the other machine has sent anything. +func (env *cliEnv) awaitPending(ctx context.Context, client controlClient, budget time.Duration) (string, int) { + deadline := env.now().Add(budget) + for { + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:pending", nil) + cancel() + if err != nil { + return "", env.fail(err) + } + var result struct { + Invites []pairing.Invite `json:"invites"` + } + if err := json.Unmarshal(raw, &result); err != nil { + return "", env.fail(fmt.Errorf("decode the pending invites: %w", err)) + } + switch len(result.Invites) { + case 0: + // keep waiting + case 1: + return result.Invites[0].InviteID, exitOK + default: + // Let the endpoint produce the message that names them all, so + // the wording lives in one place. + return "", exitOK + } + if !env.now().Before(deadline) { + fmt.Fprintf(env.errOut, "nvpair-tui accept: no invitation arrived within %s\n", budget) + return "", exitFailure + } + select { + case <-ctx.Done(): + return "", env.fail(ctx.Err()) + case <-time.After(env.poll): + } + } +} + +// --- members ---------------------------------------------------------------- + +func cmdMembers(env *cliEnv, args []string) int { + fs := flag.NewFlagSet("members", flag.ContinueOnError) + fs.SetOutput(env.errOut) + common := registerCommon(fs) + positional, err := parseArgs(fs, args) + if err != nil { + return exitFailure + } + if len(positional) != 0 { + fmt.Fprintln(env.errOut, "nvpair-tui members: takes no arguments") + return exitFailure + } + return env.callWithClient(*common.socket, func(ctx context.Context, client controlClient) int { + callCtx, cancel := context.WithTimeout(ctx, dialTimeout) + raw, err := client.Call(callCtx, "pair:members", nil) + cancel() + if err != nil { + return env.fail(err) + } + var membership pairing.Membership + if err := json.Unmarshal(raw, &membership); err != nil { + return env.fail(fmt.Errorf("decode the membership: %w", err)) + } + return env.emit(*common.json, raw, func() { + cluster := membership.ClusterID + if cluster == "" { + cluster = "(none - this machine is not in a cluster)" + } else if membership.ClusterFriendlyName != "" { + cluster += " " + membership.ClusterFriendlyName + } + fmt.Fprintf(env.out, "cluster %s\n", cluster) + if len(membership.Members) == 0 { + fmt.Fprintln(env.out, "no members") + return + } + members := append([]pairing.ClusterNode(nil), membership.Members...) + sort.SliceStable(members, func(i, j int) bool { return members[i].ID < members[j].ID }) + for _, node := range members { + fmt.Fprintf(env.out, "%s %s %s:%d %s\n", node.ID, node.Name, node.IPAddress, node.Port, node.State) + } + }, exitOK) + }) +} + +// --- shared rendering ------------------------------------------------------- + +func inviterName(inv pairing.Invite) string { + switch { + case inv.FromNodeName != "": + return inv.FromNodeName + case inv.FromNodeID != "": + return inv.FromNodeID + default: + return "the inviting machine" + } +} + +// reasonText turns a machine-readable reason into something an operator reads. +func reasonText(reason string) string { + switch reason { + case "": + return "no reason given" + case "already-clustered": + return "it is already in a cluster" + case pairing.ReasonIncorrectPin: + return "the PIN was wrong" + default: + return reason + } +} + +func reasonSuffix(reason string) string { + if reason == "" { + return "" + } + return " (" + reasonText(reason) + ")" +} diff --git a/services/nvpair-tui/cli_test.go b/services/nvpair-tui/cli_test.go new file mode 100644 index 00000000..abd3f688 --- /dev/null +++ b/services/nvpair-tui/cli_test.go @@ -0,0 +1,642 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "io" + "strings" + "sync" + "testing" + "time" + + "nvpair-tui/control" +) + +const cliTestPIN = "402199" + +// fakeEndpoint stands in for a running TUI's control socket, recording what +// each subcommand asked for and answering with whatever the test scripted. +type fakeEndpoint struct { + mu sync.Mutex + calls []endpointCall + answers map[string][]answer + dialErr error +} + +type endpointCall struct { + method string + params map[string]any +} + +type answer struct { + result string + err error +} + +func newFakeEndpoint() *fakeEndpoint { + return &fakeEndpoint{answers: map[string][]answer{}} +} + +// on queues one answer for method. Queued answers are consumed in order, so a +// polling loop can be given a sequence; the last one repeats. +func (f *fakeEndpoint) on(method, result string) *fakeEndpoint { + f.mu.Lock() + defer f.mu.Unlock() + f.answers[method] = append(f.answers[method], answer{result: result}) + return f +} + +func (f *fakeEndpoint) onError(method string, err error) *fakeEndpoint { + f.mu.Lock() + defer f.mu.Unlock() + f.answers[method] = append(f.answers[method], answer{err: err}) + return f +} + +func (f *fakeEndpoint) Call(_ context.Context, method string, params any) (json.RawMessage, error) { + f.mu.Lock() + defer f.mu.Unlock() + recorded := endpointCall{method: method} + if m, ok := params.(map[string]any); ok { + recorded.params = m + } + f.calls = append(f.calls, recorded) + + queued := f.answers[method] + if len(queued) == 0 { + return nil, fmt.Errorf("no scripted answer for %q", method) + } + next := queued[0] + if len(queued) > 1 { + f.answers[method] = queued[1:] + } + if next.err != nil { + return nil, next.err + } + return json.RawMessage(next.result), nil +} + +func (f *fakeEndpoint) Close() error { return nil } + +func (f *fakeEndpoint) paramsFor(t *testing.T, method string) map[string]any { + t.Helper() + f.mu.Lock() + defer f.mu.Unlock() + for _, c := range f.calls { + if c.method == method { + return c.params + } + } + t.Fatalf("%s was never called; calls = %+v", method, f.calls) + return nil +} + +func (f *fakeEndpoint) countOf(method string) int { + f.mu.Lock() + defer f.mu.Unlock() + n := 0 + for _, c := range f.calls { + if c.method == method { + n++ + } + } + return n +} + +// run executes one subcommand against the fake endpoint and reports its exit +// code together with what it wrote. +type runResult struct { + code int + stdout string + stderr string +} + +func (r runResult) all() string { return r.stdout + r.stderr } + +func run(t *testing.T, endpoint *fakeEndpoint, env map[string]string, args ...string) runResult { + t.Helper() + var out, errOut bytes.Buffer + // A fixed clock keeps the age column and every --wait deadline + // deterministic; the poll loops still advance it through fakeClock. + clock := &fakeClock{now: time.UnixMilli(1716998460000)} + cli := &cliEnv{ + out: &out, + errOut: &errOut, + getenv: func(k string) string { return env[k] }, + now: clock.Now, + poll: time.Millisecond, + dial: func(string) (controlClient, error) { + if endpoint.dialErr != nil { + return nil, endpoint.dialErr + } + return endpoint, nil + }, + } + code := runSubcommand(cli, args) + return runResult{code: code, stdout: out.String(), stderr: errOut.String()} +} + +// fakeClock advances a little on every read, so a polling loop bounded by a +// deadline terminates without the test sleeping. +type fakeClock struct { + mu sync.Mutex + now time.Time +} + +func (c *fakeClock) Now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + c.now = c.now.Add(time.Second) + return c.now +} + +func TestSubcommandNameOnlyMatchesAVerb(t *testing.T) { + tests := []struct { + args []string + want string + }{ + {nil, ""}, + {[]string{}, ""}, + {[]string{"--version"}, ""}, + {[]string{"-log-level", "debug"}, ""}, + {[]string{"--control-socket", "/tmp/x.sock"}, ""}, + {[]string{"invite", "10.0.0.5"}, "invite"}, + {[]string{"members"}, "members"}, + {[]string{"bogus"}, "bogus"}, + } + for _, tc := range tests { + if got := subcommandName(tc.args); got != tc.want { + t.Errorf("subcommandName(%v) = %q, want %q", tc.args, got, tc.want) + } + } +} + +func TestParseArgsAcceptsFlagsAroundThePositional(t *testing.T) { + for _, args := range [][]string{ + {"10.0.0.5", "--port", "14399", "--json"}, + {"--port", "14399", "10.0.0.5", "--json"}, + {"--json", "--port", "14399", "10.0.0.5"}, + } { + t.Run(strings.Join(args, " "), func(t *testing.T) { + fs := flag.NewFlagSet("t", flag.ContinueOnError) + fs.SetOutput(io.Discard) + port := fs.Int("port", 0, "") + asJSON := fs.Bool("json", false, "") + positional, err := parseArgs(fs, args) + if err != nil { + t.Fatalf("parseArgs: %v", err) + } + if len(positional) != 1 || positional[0] != "10.0.0.5" { + t.Errorf("positional = %v, want the address", positional) + } + if *port != 14399 || !*asJSON { + t.Errorf("port = %d, json = %v; flags on either side of the positional must all bind", *port, *asJSON) + } + }) + } +} + +func TestUnknownCommandPrintsUsage(t *testing.T) { + got := run(t, newFakeEndpoint(), nil, "bogus") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + for _, want := range []string{`unknown command "bogus"`, "nvpair-tui invite", "Exit codes"} { + if !strings.Contains(got.stderr, want) { + t.Errorf("stderr does not mention %q:\n%s", want, got.stderr) + } + } +} + +func TestNoRunningInstanceExitsWithAnExplanation(t *testing.T) { + endpoint := newFakeEndpoint() + endpoint.dialErr = &control.NotRunningError{Path: "/run/nvpair/tui.sock", Err: fmt.Errorf("connect: no such file or directory")} + + for _, args := range [][]string{{"members"}, {"pending"}, {"invite", "10.0.0.5"}, {"decline"}} { + t.Run(args[0], func(t *testing.T) { + got := run(t, endpoint, map[string]string{pinEnvVar: cliTestPIN}, args...) + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + if !strings.Contains(got.stderr, "no nvpair-tui is listening") { + t.Errorf("stderr does not say nothing is listening:\n%s", got.stderr) + } + if !strings.Contains(got.stderr, "Start the terminal interface") { + t.Errorf("stderr does not say what to do about it:\n%s", got.stderr) + } + }) + } +} + +func TestInvitePrintsThePINOnStdout(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:invite", + `{"inviteId":"inv-1","state":"pending","pin":"`+cliTestPIN+`"}`) + + got := run(t, endpoint, nil, "invite", "10.0.0.5", "--port", "14399") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if !strings.Contains(got.stdout, cliTestPIN) { + t.Errorf("the PIN is not on stdout:\n%s", got.stdout) + } + if !strings.Contains(got.stdout, "inv-1") { + t.Errorf("the invite id is not on stdout:\n%s", got.stdout) + } + params := endpoint.paramsFor(t, "pair:invite") + if params["address"] != "10.0.0.5" || params["port"] != 14399 { + t.Errorf("params = %v, want the address and the port asked for", params) + } +} + +func TestInviteWithoutAPortSendsNone(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:invite", `{"inviteId":"inv-1","state":"pending","pin":"`+cliTestPIN+`"}`) + run(t, endpoint, nil, "invite", "gpu-box.tail1234.ts.net") + params := endpoint.paramsFor(t, "pair:invite") + if _, ok := params["port"]; ok { + t.Errorf("params = %v, want no port so the manager appends its own", params) + } +} + +func TestInviteJSONPrintsTheRawResult(t *testing.T) { + raw := `{"inviteId":"inv-1","state":"pending","pin":"` + cliTestPIN + `","unmodelled":true}` + endpoint := newFakeEndpoint().on("pair:invite", raw) + + got := run(t, endpoint, nil, "invite", "10.0.0.5", "--json") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if strings.TrimSpace(got.stdout) != raw { + t.Errorf("stdout = %q, want the endpoint's raw result", got.stdout) + } +} + +func TestInviteRequiresExactlyOneAddress(t *testing.T) { + for _, args := range [][]string{{"invite"}, {"invite", "a", "b"}} { + got := run(t, newFakeEndpoint(), nil, args...) + if got.code != exitFailure { + t.Errorf("%v: exit = %d, want %d", args, got.code, exitFailure) + } + if !strings.Contains(got.stderr, "exactly one address") { + t.Errorf("%v: stderr = %q", args, got.stderr) + } + } +} + +func TestInviteReportsARejection(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:invite", + `{"inviteId":"inv-1","state":"rejected","reason":"already-clustered","pin":null}`) + + got := run(t, endpoint, nil, "invite", "10.0.0.5") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + if !strings.Contains(got.stderr, "already in a cluster") { + t.Errorf("stderr does not explain the rejection:\n%s", got.stderr) + } +} + +func TestInviteWaitExitsOnTheFinalState(t *testing.T) { + tests := []struct { + name string + status string + want int + }{ + {"paired", `{"inviteId":"inv-1","state":"paired"}`, exitOK}, + {"declined", `{"inviteId":"inv-1","state":"declined"}`, exitRefused}, + {"wrong pin", `{"inviteId":"inv-1","state":"failed","reason":"incorrect-pin"}`, exitRefused}, + {"unreachable", `{"inviteId":"inv-1","state":"failed"}`, exitFailure}, + {"expired", `{"inviteId":"inv-1","state":"expired"}`, exitFailure}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + endpoint := newFakeEndpoint(). + on("pair:invite", `{"inviteId":"inv-1","state":"pending","pin":"`+cliTestPIN+`"}`). + on("pair:invite-status", tc.status) + + got := run(t, endpoint, nil, "invite", "10.0.0.5", "--wait") + if got.code != tc.want { + t.Errorf("exit = %d, want %d (%s)", got.code, tc.want, got.all()) + } + if !strings.Contains(got.stdout, cliTestPIN) { + t.Errorf("--wait must still print the PIN so it can be read out:\n%s", got.stdout) + } + }) + } +} + +// TestInviteWaitJSONPrintsOnlyJSON checks that --json means machine-readable +// output all the way through: two documents, one per line, and no prose +// mixed in for a parser to trip over. +func TestInviteWaitJSONPrintsOnlyJSON(t *testing.T) { + created := `{"inviteId":"inv-1","state":"pending","pin":"` + cliTestPIN + `"}` + final := `{"inviteId":"inv-1","state":"paired"}` + endpoint := newFakeEndpoint().on("pair:invite", created).on("pair:invite-status", final) + + got := run(t, endpoint, nil, "invite", "10.0.0.5", "--wait", "--json") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + lines := strings.Split(strings.TrimSpace(got.stdout), "\n") + if len(lines) != 2 { + t.Fatalf("stdout = %q, want the created invite and its final state, one per line", got.stdout) + } + if lines[0] != created { + t.Errorf("first line = %q, want the invite as created (it carries the PIN)", lines[0]) + } + if lines[1] != final { + t.Errorf("second line = %q, want the final invite", lines[1]) + } + for _, line := range lines { + var probe map[string]any + if err := json.Unmarshal([]byte(line), &probe); err != nil { + t.Errorf("line %q is not JSON: %v", line, err) + } + } + if got.stderr != "" { + t.Errorf("stderr = %q, want nothing on a successful --json run", got.stderr) + } +} + +func TestInviteWaitPollsUntilTheInviteLeavesPending(t *testing.T) { + endpoint := newFakeEndpoint(). + on("pair:invite", `{"inviteId":"inv-1","state":"pending","pin":"`+cliTestPIN+`"}`). + on("pair:invite-status", `{"inviteId":"inv-1","state":"pending"}`). + on("pair:invite-status", `{"inviteId":"inv-1","state":"pending"}`). + on("pair:invite-status", `{"inviteId":"inv-1","state":"paired"}`) + + got := run(t, endpoint, nil, "invite", "10.0.0.5", "--wait") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if n := endpoint.countOf("pair:invite-status"); n != 3 { + t.Errorf("polled %d times, want it to keep asking until the invite settled", n) + } +} + +func TestAcceptExitCodes(t *testing.T) { + tests := []struct { + name string + response string + want int + }{ + {"paired", `{"inviteId":"inv-1","state":"paired","fromNodeName":"Lab desk A"}`, exitOK}, + {"wrong pin", `{"inviteId":"inv-1","state":"failed","reason":"incorrect-pin"}`, exitRefused}, + {"expired", `{"inviteId":"inv-1","state":"expired"}`, exitFailure}, + {"other failure", `{"inviteId":"inv-1","state":"failed"}`, exitFailure}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", tc.response) + got := run(t, endpoint, nil, "accept", "--pin", cliTestPIN) + if got.code != tc.want { + t.Errorf("exit = %d, want %d (%s)", got.code, tc.want, got.all()) + } + }) + } +} + +func TestAcceptReadsThePINFromTheEnvironment(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-1","state":"paired"}`) + got := run(t, endpoint, map[string]string{pinEnvVar: " " + cliTestPIN + " "}, "accept") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + params := endpoint.paramsFor(t, "pair:respond") + if params["pin"] != cliTestPIN { + t.Errorf("pin = %v, want the environment value trimmed and forwarded", params["pin"]) + } + if params["accept"] != true { + t.Errorf("accept = %v, want true", params["accept"]) + } +} + +func TestAcceptWithoutAPINSaysWhereToPutOne(t *testing.T) { + got := run(t, newFakeEndpoint(), nil, "accept") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + if !strings.Contains(got.stderr, pinEnvVar) { + t.Errorf("stderr does not point at %s:\n%s", pinEnvVar, got.stderr) + } +} + +func TestAcceptPassesAnExplicitInviteID(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-2","state":"paired"}`) + run(t, endpoint, nil, "accept", "--pin", cliTestPIN, "--invite", "inv-2") + if got := endpoint.paramsFor(t, "pair:respond")["inviteId"]; got != "inv-2" { + t.Errorf("inviteId = %v, want the one named", got) + } +} + +func TestAcceptOmitsTheInviteIDSoTheEndpointResolvesIt(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-1","state":"paired"}`) + run(t, endpoint, nil, "accept", "--pin", cliTestPIN) + if _, ok := endpoint.paramsFor(t, "pair:respond")["inviteId"]; ok { + t.Error("an unnamed invite must be resolved by the endpoint, not guessed here") + } +} + +func TestAcceptSurfacesTheAmbiguousInviteError(t *testing.T) { + endpoint := newFakeEndpoint().onError("pair:respond", &control.Error{ + Code: control.CodeAmbiguousInvite, + Message: "2 invites are pending; name one with --invite: inv-1 (from Lab desk A), inv-2 (from Lab desk B)", + }) + got := run(t, endpoint, nil, "accept", "--pin", cliTestPIN) + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + for _, want := range []string{"inv-1", "inv-2", "--invite"} { + if !strings.Contains(got.stderr, want) { + t.Errorf("stderr does not mention %q:\n%s", want, got.stderr) + } + } +} + +func TestAcceptWaitsForAnInviteToArrive(t *testing.T) { + endpoint := newFakeEndpoint(). + on("pair:pending", `{"invites":[]}`). + on("pair:pending", `{"invites":[]}`). + on("pair:pending", `{"invites":[{"inviteId":"inv-late","fromNodeName":"Lab desk A","state":"pending"}]}`). + on("pair:respond", `{"inviteId":"inv-late","state":"paired"}`) + + got := run(t, endpoint, nil, "accept", "--pin", cliTestPIN, "--wait", "2m") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if id := endpoint.paramsFor(t, "pair:respond")["inviteId"]; id != "inv-late" { + t.Errorf("inviteId = %v, want the invite that arrived while waiting", id) + } +} + +func TestAcceptGivesUpWhenNoInviteArrives(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:pending", `{"invites":[]}`) + got := run(t, endpoint, nil, "accept", "--pin", cliTestPIN, "--wait", "5s") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + if !strings.Contains(got.stderr, "no invitation arrived") { + t.Errorf("stderr = %q, want it to say the wait ran out", got.stderr) + } + if endpoint.countOf("pair:respond") != 0 { + t.Error("a timed-out wait must not go on to answer anything") + } +} + +func TestAcceptRejectsAStrayArgument(t *testing.T) { + got := run(t, newFakeEndpoint(), nil, "accept", "--pin", cliTestPIN, "402199") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } + if !strings.Contains(got.stderr, "unexpected argument") { + t.Errorf("stderr = %q", got.stderr) + } +} + +func TestDeclineExitCodes(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-1","state":"declined"}`) + got := run(t, endpoint, nil, "decline") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + params := endpoint.paramsFor(t, "pair:respond") + if params["accept"] != false { + t.Errorf("accept = %v, want false", params["accept"]) + } + if _, ok := params["pin"]; ok { + t.Error("a decline sent a pin; there is nothing to prove on a decline") + } +} + +func TestDeclineReportsAnUnexpectedState(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-1","state":"expired"}`) + got := run(t, endpoint, nil, "decline") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } +} + +func TestPendingListsInvitesWithTheirAge(t *testing.T) { + // receivedAt is 30 s before the clock the fake environment starts on. + endpoint := newFakeEndpoint(). + on("pair:pending", `{"invites":[{"inviteId":"inv-1","fromNodeName":"Lab desk A","fromNodeUuid":"uuid-a","state":"pending","receivedAt":1716998430000}]}`). + on("pair:members", `{"clusterId":"c","members":[{"id":"NODE-A","nodeUuid":"uuid-a","name":"Lab desk A","ipAddress":"10.0.0.5","port":14321,"state":"pending-inbound"}]}`) + + got := run(t, endpoint, nil, "pending") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + for _, want := range []string{"inv-1", "Lab desk A", "10.0.0.5", "ago"} { + if !strings.Contains(got.stdout, want) { + t.Errorf("stdout does not show %q:\n%s", want, got.stdout) + } + } +} + +func TestPendingWithNothingWaiting(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:pending", `{"invites":[]}`) + got := run(t, endpoint, nil, "pending") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if !strings.Contains(got.stdout, "No invitations") { + t.Errorf("stdout = %q", got.stdout) + } + if endpoint.countOf("pair:members") != 0 { + t.Error("an empty list must not go looking up addresses") + } +} + +func TestPendingStillListsWhenTheAddressLookupFails(t *testing.T) { + endpoint := newFakeEndpoint(). + on("pair:pending", `{"invites":[{"inviteId":"inv-1","fromNodeName":"Lab desk A","state":"pending","receivedAt":1716998430000}]}`). + onError("pair:members", fmt.Errorf("cluster manager unavailable")) + + got := run(t, endpoint, nil, "pending") + if got.code != exitOK { + t.Fatalf("exit = %d (%s); the address is a convenience, not a requirement", got.code, got.all()) + } + if !strings.Contains(got.stdout, "inv-1") { + t.Errorf("stdout = %q", got.stdout) + } +} + +func TestMembersPrintsTheClusterAndItsRoster(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:members", + `{"clusterId":"cluster-xyz","clusterFriendlyName":"Lab 3 desks","nodeId":"NODE-A","members":[`+ + `{"id":"NODE-B","name":"Lab desk B","ipAddress":"10.0.0.5","port":14321,"state":"member"},`+ + `{"id":"NODE-A","name":"Lab desk A","ipAddress":"10.0.0.4","port":14321,"state":"member"}]}`) + + got := run(t, endpoint, nil, "members") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + for _, want := range []string{"cluster-xyz", "Lab 3 desks", "NODE-A", "NODE-B", "10.0.0.5"} { + if !strings.Contains(got.stdout, want) { + t.Errorf("stdout does not show %q:\n%s", want, got.stdout) + } + } + if strings.Index(got.stdout, "NODE-A") > strings.Index(got.stdout, "NODE-B") { + t.Errorf("members are not in a stable order:\n%s", got.stdout) + } +} + +func TestMembersOfAnUnclusteredNode(t *testing.T) { + endpoint := newFakeEndpoint().on("pair:members", `{"clusterId":"","members":[]}`) + got := run(t, endpoint, nil, "members") + if got.code != exitOK { + t.Fatalf("exit = %d (%s)", got.code, got.all()) + } + if !strings.Contains(got.stdout, "not in a cluster") { + t.Errorf("stdout = %q", got.stdout) + } +} + +func TestMembersRejectsAnArgument(t *testing.T) { + got := run(t, newFakeEndpoint(), nil, "members", "extra") + if got.code != exitFailure { + t.Errorf("exit = %d, want %d", got.code, exitFailure) + } +} + +// TestNoSubcommandLeaksThePINIntoItsOwnDiagnostics guards the one thing that +// must never travel: every failure path is exercised with a PIN in hand, and +// none of the messages may repeat it back except the deliberate invite line +// that exists to be read out loud. +func TestNoSubcommandLeaksThePINIntoItsOwnDiagnostics(t *testing.T) { + env := map[string]string{pinEnvVar: cliTestPIN} + cases := []struct { + name string + endpoint *fakeEndpoint + args []string + }{ + { + name: "a wrong PIN", + endpoint: newFakeEndpoint().on("pair:respond", `{"inviteId":"inv-1","state":"failed","reason":"incorrect-pin"}`), + args: []string{"accept"}, + }, + { + name: "nothing pending", + endpoint: newFakeEndpoint().onError("pair:respond", &control.Error{Code: control.CodeNoPendingInvite, Message: "no invite is pending on this node"}), + args: []string{"accept"}, + }, + { + name: "a transport failure", + endpoint: newFakeEndpoint().onError("pair:respond", fmt.Errorf("pair:respond: connection closed")), + args: []string{"accept"}, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := run(t, tc.endpoint, env, tc.args...) + if strings.Contains(got.all(), cliTestPIN) { + t.Errorf("the PIN was echoed back:\nstdout=%s\nstderr=%s", got.stdout, got.stderr) + } + }) + } +} diff --git a/services/nvpair-tui/control/client.go b/services/nvpair-tui/control/client.go new file mode 100644 index 00000000..7cbdb8ab --- /dev/null +++ b/services/nvpair-tui/control/client.go @@ -0,0 +1,93 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package control + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + + "nvpair-shared/ipc" + "nvpair-shared/jsonrpc" +) + +// ErrNotRunning is the "no nvpair-tui is running on this machine" case, which +// every subcommand has to report differently from a request that reached a +// running TUI and failed there. +var ErrNotRunning = errors.New("no running nvpair-tui was found") + +// NotRunningError carries ErrNotRunning together with the endpoint that was +// tried and the reason it could not be reached, so the message can name both. +type NotRunningError struct { + Path string + Err error +} + +func (e *NotRunningError) Error() string { + return fmt.Sprintf("no nvpair-tui is listening on %s (%v)", e.Path, e.Err) +} + +func (e *NotRunningError) Unwrap() error { return ErrNotRunning } + +// Error is a JSON-RPC error the control endpoint returned. Where the request +// was relayed to nvpair-cluster-manager, Code is the manager's own, so a +// caller can key on its documented contract through this endpoint. +type Error struct { + Code int `json:"code"` + Message string `json:"message"` + Data json.RawMessage `json:"data,omitempty"` +} + +func (e *Error) Error() string { return e.Message } + +// Client is one connection to a running TUI's control endpoint. +type Client struct { + conn io.Closer + peer *jsonrpc.Peer +} + +// Dial connects to the control endpoint at path. A refused or missing +// endpoint means no TUI is running there and comes back as *NotRunningError +// rather than a bare transport error. +func Dial(path string) (*Client, error) { + conn, err := ipc.Dial(path) + if err != nil { + return nil, &NotRunningError{Path: path, Err: err} + } + peer := jsonrpc.NewPeer(jsonrpc.NewCodec(conn)) + // The endpoint never sends requests or notifications, so the pump only + // has to wake Call waiters. + go peer.Serve(nil, nil) + return &Client{conn: conn, peer: peer}, nil +} + +// Close ends the session. Closing the transport also stops the read pump. +func (c *Client) Close() error { + c.peer.Close() + return c.conn.Close() +} + +// Call issues one request and returns its raw result. A JSON-RPC error +// response comes back as *Error; anything that stopped the call from +// completing at all comes back as a plain error. +func (c *Client) Call(ctx context.Context, method string, params any) (json.RawMessage, error) { + var encoded json.RawMessage + if params != nil { + b, err := json.Marshal(params) + if err != nil { + return nil, fmt.Errorf("encode %s params: %w", method, err) + } + encoded = b + } + result, rpcErr, err := c.peer.Call(ctx, method, encoded) + if err != nil { + return nil, fmt.Errorf("%s: %w", method, err) + } + if rpcErr != nil { + return nil, &Error{Code: rpcErr.Code, Message: rpcErr.Message, Data: rpcErr.Data} + } + return result, nil +} diff --git a/services/nvpair-tui/control/endpoint_unix_test.go b/services/nvpair-tui/control/endpoint_unix_test.go new file mode 100644 index 00000000..86aee161 --- /dev/null +++ b/services/nvpair-tui/control/endpoint_unix_test.go @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build !windows + +package control + +import ( + "path/filepath" + "testing" +) + +// socketPath gives one test its own endpoint, short enough to stay inside the +// platform's sun_path limit. +func socketPath(t *testing.T) string { + t.Helper() + return filepath.Join(shortTempDir(t), "tui.sock") +} diff --git a/services/nvpair-tui/control/endpoint_windows_test.go b/services/nvpair-tui/control/endpoint_windows_test.go new file mode 100644 index 00000000..41f86b95 --- /dev/null +++ b/services/nvpair-tui/control/endpoint_windows_test.go @@ -0,0 +1,20 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build windows + +package control + +import ( + "fmt" + "os" + "testing" +) + +// socketPath gives one test its own named pipe. The pipe namespace is +// machine-wide and flat, so the name carries the pid and the test name to keep +// parallel runs apart. +func socketPath(t *testing.T) string { + t.Helper() + return fmt.Sprintf(`\\.\pipe\nvpair-tui-test-%d-%s`, os.Getpid(), sanitize(t.Name())) +} diff --git a/services/nvpair-tui/control/path.go b/services/nvpair-tui/control/path.go new file mode 100644 index 00000000..a73acfc1 --- /dev/null +++ b/services/nvpair-tui/control/path.go @@ -0,0 +1,94 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Package control is nvpair-tui's local control endpoint: a per-user socket a +// running TUI listens on so a second invocation of the same binary — or any +// script on the box — can drive pairing without a keyboard. +// +// It speaks the same newline-delimited JSON-RPC 2.0 every other PAIR surface +// speaks, and every method it exposes is relayed through the TUI's existing +// broker connection. The broker keeps exactly one client, which is what it is +// built for; this socket does not open a second one. +// +// It is a *local* endpoint and carries no authentication of its own: on +// non-Windows the parent directory is 0700 and the socket 0600, so the file +// permissions are the check, and on Windows the named pipe's default DACL +// grants the creating user. Anything that can read the socket can pair this +// machine with another, which is the same authority the operator sitting at +// the TUI already has. +package control + +import ( + "errors" + "fmt" + "net" + "os" + "path/filepath" + + "nvpair-shared/ipc" +) + +// dirPerm / socketPerm keep the endpoint to its owner. The directory is the +// real guard: net.Listen creates the socket under the process umask, so there +// is a moment before the chmod when the mode is wider than 0600, and a 0700 +// parent makes that moment unreachable. +const ( + dirPerm os.FileMode = 0o700 + socketPerm os.FileMode = 0o600 +) + +// DefaultPath returns the well-known per-user control endpoint. +// +// On Linux with $XDG_RUNTIME_DIR set that is $XDG_RUNTIME_DIR/nvpair/tui.sock, +// the directory the OS already keeps per-user, 0700 and cleaned at logout. +// Everywhere else it is "run/tui.sock" under the shared per-user data +// directory every PAIR component agrees on (nvpair-shared/appdir). On Windows +// it is a named pipe rather than a filesystem path. +func DefaultPath() (string, error) { return defaultPath() } + +// Listen opens the control endpoint at path, creating its parent directory +// 0700 and tightening the socket to 0600. +// +// A socket file left behind by a TUI that did not exit cleanly is removed, but +// only after confirming nothing answers on it: a live socket means another TUI +// is running and this one must not steal its endpoint. +func Listen(path string) (net.Listener, error) { + if path == "" { + return nil, errors.New("control socket path is empty") + } + if err := prepare(path); err != nil { + return nil, err + } + ln, err := ipc.Listen(path) + if err != nil { + return nil, fmt.Errorf("listen on control socket %s: %w", path, err) + } + if err := secure(path); err != nil { + _ = ln.Close() + return nil, err + } + return ln, nil +} + +// InUse reports whether something is already answering at path, so a caller +// can tell "no TUI is running" from "the endpoint is taken". +func InUse(path string) bool { + conn, err := ipc.Dial(path) + if err != nil { + return false + } + _ = conn.Close() + return true +} + +// ensureDir creates the endpoint's parent directory 0700. An existing +// directory keeps whatever mode it has: on Linux $XDG_RUNTIME_DIR is already +// 0700 and owned by the user, and re-chmod'ing a directory PAIR does not own +// is not this component's business. +func ensureDir(path string) error { + dir := filepath.Dir(path) + if err := os.MkdirAll(dir, dirPerm); err != nil { + return fmt.Errorf("create control socket directory %s: %w", dir, err) + } + return nil +} diff --git a/services/nvpair-tui/control/path_unix.go b/services/nvpair-tui/control/path_unix.go new file mode 100644 index 00000000..cde35ccd --- /dev/null +++ b/services/nvpair-tui/control/path_unix.go @@ -0,0 +1,84 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build !windows + +package control + +import ( + "fmt" + "os" + "path/filepath" + + "nvpair-shared/appdir" +) + +// maxSocketPath is the shortest sun_path any platform PAIR builds for allows: +// 104 bytes on the BSDs and macOS, 108 on Linux. A path at or over the limit +// is silently truncated by the kernel, which produces a socket at a +// nonsensical path rather than an error, so it is checked up front and +// reported with the flag that fixes it. +const maxSocketPath = 104 + +// defaultPath prefers the OS's own per-user runtime directory, which exists +// precisely for sockets and is already 0700, and otherwise puts the socket in +// a run/ subdirectory of the shared per-user data directory. +func defaultPath() (string, error) { + if runtimeDir := os.Getenv("XDG_RUNTIME_DIR"); runtimeDir != "" { + return check(filepath.Join(runtimeDir, "nvpair", "tui.sock")) + } + path, err := appdir.Path("run", "tui.sock") + if err != nil { + return "", fmt.Errorf("resolve the per-user data directory: %w", err) + } + return check(path) +} + +// check rejects a path the kernel would truncate, naming the escape hatch. A +// truncated sun_path is not an error the kernel reports as one: the bind +// either fails with a bare EINVAL or, worse, succeeds at a path nobody will +// dial, so the length is checked before either can happen. +func check(path string) (string, error) { + if len(path) >= maxSocketPath { + return "", fmt.Errorf( + "the control socket path is %d bytes, over this platform's %d-byte limit (%s); "+ + "pass --control-socket with a shorter path, or --no-control-socket to run without one", + len(path), maxSocketPath, path) + } + return path, nil +} + +// prepare makes the endpoint's directory and clears a socket file left by a +// TUI that did not exit cleanly. A file that still answers belongs to a +// running TUI and is never removed. +func prepare(path string) error { + if _, err := check(path); err != nil { + return err + } + if err := ensureDir(path); err != nil { + return err + } + if _, err := os.Lstat(path); err != nil { + if os.IsNotExist(err) { + return nil + } + return fmt.Errorf("inspect control socket %s: %w", path, err) + } + if InUse(path) { + return fmt.Errorf("another nvpair-tui is already listening on %s", path) + } + if err := os.Remove(path); err != nil { + return fmt.Errorf("remove the stale control socket %s: %w", path, err) + } + return nil +} + +// secure narrows the socket to its owner. net.Listen created it under the +// process umask, which on a default umask is already 0755 — wide enough for +// another local user to connect were the parent directory not 0700. +func secure(path string) error { + if err := os.Chmod(path, socketPerm); err != nil { + return fmt.Errorf("restrict the control socket %s to its owner: %w", path, err) + } + return nil +} diff --git a/services/nvpair-tui/control/path_unix_test.go b/services/nvpair-tui/control/path_unix_test.go new file mode 100644 index 00000000..39bfade1 --- /dev/null +++ b/services/nvpair-tui/control/path_unix_test.go @@ -0,0 +1,173 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build !windows + +package control + +import ( + "net" + "os" + "path/filepath" + "strings" + "testing" +) + +// shortTempDir gives a temp directory whose path leaves room for a socket +// name. t.TempDir() on macOS sits under /var/folders/... and, once the +// per-user data directory is appended, can pass the 104-byte sun_path limit — +// which is a real constraint of this endpoint, not an artefact of the test. +func shortTempDir(t *testing.T) string { + t.Helper() + dir, err := os.MkdirTemp("/tmp", "nvctl") + if err != nil { + t.Fatalf("temp dir: %v", err) + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) + return dir +} + +func TestDefaultPathPrefersTheRuntimeDirectory(t *testing.T) { + runtimeDir := shortTempDir(t) + t.Setenv("XDG_RUNTIME_DIR", runtimeDir) + + got, err := DefaultPath() + if err != nil { + t.Fatalf("DefaultPath: %v", err) + } + want := filepath.Join(runtimeDir, "nvpair", "tui.sock") + if got != want { + t.Errorf("DefaultPath() = %q, want %q", got, want) + } +} + +func TestDefaultPathFallsBackToThePerUserDataDirectory(t *testing.T) { + base := shortTempDir(t) + t.Setenv("XDG_RUNTIME_DIR", "") + t.Setenv("HOME", base) + t.Setenv("XDG_CONFIG_HOME", base) + + got, err := DefaultPath() + if err != nil { + t.Fatalf("DefaultPath: %v", err) + } + // The tail is what matters: every PAIR component agrees on the vendor and + // product directories, and this endpoint lives in a run/ subdirectory + // beneath them. + wantTail := filepath.Join("Nvidia Corporation", "Personal AI Router", "run", "tui.sock") + if !strings.HasSuffix(got, wantTail) { + t.Errorf("DefaultPath() = %q, want it to end in %q", got, wantTail) + } + if !strings.HasPrefix(got, base) { + t.Errorf("DefaultPath() = %q, want it under the per-user base %q", got, base) + } +} + +func TestListenRejectsAPathTheKernelWouldTruncate(t *testing.T) { + long := filepath.Join("/tmp", strings.Repeat("d", maxSocketPath), "tui.sock") + _, err := Listen(long) + if err == nil { + t.Fatal("Listen accepted a path past the sun_path limit") + } + for _, want := range []string{"limit", "--control-socket", "--no-control-socket"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q does not mention %q", err, want) + } + } + if _, err := os.Stat(filepath.Dir(long)); err == nil { + t.Error("Listen created a directory for a path it was going to reject") + } +} + +func TestListenCreatesAPrivateEndpoint(t *testing.T) { + path := filepath.Join(shortTempDir(t), "nested", "tui.sock") + ln, err := Listen(path) + if err != nil { + t.Fatalf("Listen: %v", err) + } + defer func() { _ = ln.Close() }() + + info, err := os.Stat(path) + if err != nil { + t.Fatalf("stat the socket: %v", err) + } + if mode := info.Mode().Perm(); mode != socketPerm { + t.Errorf("socket mode = %o, want %o: only its owner may drive pairing", mode, socketPerm) + } + dirInfo, err := os.Stat(filepath.Dir(path)) + if err != nil { + t.Fatalf("stat the directory: %v", err) + } + if mode := dirInfo.Mode().Perm(); mode != dirPerm { + t.Errorf("directory mode = %o, want %o", mode, dirPerm) + } +} + +func TestListenRefusesToStealALiveEndpoint(t *testing.T) { + path := filepath.Join(shortTempDir(t), "tui.sock") + first, err := Listen(path) + if err != nil { + t.Fatalf("first Listen: %v", err) + } + defer func() { _ = first.Close() }() + + second, err := Listen(path) + if err == nil { + _ = second.Close() + t.Fatal("a second nvpair-tui took over a socket another one is serving") + } + if !strings.Contains(err.Error(), "already listening") { + t.Errorf("error = %q, want it to say another instance holds the endpoint", err) + } + // The live endpoint must still be there and still answering. + if !InUse(path) { + t.Error("the first instance's endpoint was removed by the second's attempt") + } +} + +func TestListenReclaimsASocketLeftByADeadInstance(t *testing.T) { + path := filepath.Join(shortTempDir(t), "tui.sock") + + // A TUI that was killed leaves the socket file behind with nothing + // answering on it — SetUnlinkOnClose(false) reproduces exactly that. + stale, err := net.Listen("unix", path) + if err != nil { + t.Fatalf("create the stale socket: %v", err) + } + stale.(*net.UnixListener).SetUnlinkOnClose(false) + if err := stale.Close(); err != nil { + t.Fatalf("close the stale listener: %v", err) + } + if _, err := os.Stat(path); err != nil { + t.Fatalf("the stale socket file should still exist: %v", err) + } + + ln, err := Listen(path) + if err != nil { + t.Fatalf("Listen did not reclaim a stale socket: %v", err) + } + defer func() { _ = ln.Close() }() + if !InUse(path) { + t.Error("the reclaimed endpoint does not answer") + } +} + +func TestClosingTheListenerRemovesTheSocket(t *testing.T) { + path := filepath.Join(shortTempDir(t), "tui.sock") + ln, err := Listen(path) + if err != nil { + t.Fatalf("Listen: %v", err) + } + if err := ln.Close(); err != nil { + t.Fatalf("Close: %v", err) + } + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Errorf("the socket outlived its listener (stat err = %v)", err) + } +} + +func TestInUseIsFalseForAPathWithNoEndpoint(t *testing.T) { + if InUse(filepath.Join(shortTempDir(t), "nothing.sock")) { + t.Error("InUse reported an endpoint where there is no file at all") + } +} diff --git a/services/nvpair-tui/control/path_windows.go b/services/nvpair-tui/control/path_windows.go new file mode 100644 index 00000000..dab629e0 --- /dev/null +++ b/services/nvpair-tui/control/path_windows.go @@ -0,0 +1,53 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build windows + +package control + +import ( + "fmt" + "os/user" + "strings" +) + +// defaultPath is a named pipe rather than a file. The pipe namespace is flat +// and machine-wide, so the name carries the user's SID to keep two signed-in +// users from colliding — the same reason the Unix path is per-user. +func defaultPath() (string, error) { + u, err := user.Current() + if err != nil { + return "", fmt.Errorf("resolve the current user: %w", err) + } + // A SID ("S-1-5-21-...") is already pipe-name safe; a username may not be, + // so anything unexpected is reduced to characters a pipe name accepts. + return `\\.\pipe\nvpair-tui-` + sanitize(u.Uid), nil +} + +func sanitize(s string) string { + return strings.Map(func(r rune) rune { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_': + return r + default: + return '_' + } + }, s) +} + +// prepare has nothing to clean up: a named pipe exists only while its server +// process does, so there is no stale endpoint to remove. A pipe still held by +// a running TUI makes the listen itself fail, which is the check. +func prepare(path string) error { + if !strings.HasPrefix(path, `\\.\pipe\`) { + // An operator who passed a filesystem path with --control-socket gets + // a Unix-socket-shaped endpoint; it still needs its directory. + return ensureDir(path) + } + return nil +} + +// secure is a no-op for a named pipe: go-winio creates it with the default +// DACL, which grants the creating user and denies everyone else. A +// filesystem path passed with --control-socket has no mode to set on Windows. +func secure(path string) error { return nil } diff --git a/services/nvpair-tui/control/server.go b/services/nvpair-tui/control/server.go new file mode 100644 index 00000000..f78559c8 --- /dev/null +++ b/services/nvpair-tui/control/server.go @@ -0,0 +1,276 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package control + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "net" + "sync" + "time" + + "nvpair-shared/jsonrpc" + "nvpair-tui/pairing" + "nvpair-tui/rpc" +) + +// callTimeout bounds one relayed request. cluster:invite-node and +// cluster:respond-to-invite drive multi-round inter-node exchanges, and the +// broker allows itself 30 s for them, so this sits above that — the same +// budget the interactive UI gives its own calls. +const callTimeout = 35 * time.Second + +// JSON-RPC error codes this endpoint originates. The relayed cluster-manager +// codes come back untouched, so these deliberately sit outside its range +// (-32001 unknown invite, -32002 invalid state, -32004 precondition). +const ( + // CodeNoPendingInvite — a response was asked for with no invite id and + // nothing is waiting. + CodeNoPendingInvite = -32010 + // CodeAmbiguousInvite — a response was asked for with no invite id and + // several invites are waiting. data.invites names them. + CodeAmbiguousInvite = -32011 + // CodeInvalidParams / CodeMethodNotFound / CodeInternal are the standard + // JSON-RPC 2.0 codes. + CodeInvalidParams = -32602 + CodeMethodNotFound = -32601 + CodeInternal = -32603 + CodeParseError = -32700 +) + +// Server answers control-socket requests by relaying them through the TUI's +// one broker connection. +type Server struct { + pairing *pairing.Service + version string + ready func() bool +} + +// NewServer builds the control endpoint's request handler. version is the +// nvpair-tui build version and ready reports whether the broker has announced +// itself; both are what `ping` answers with. +func NewServer(svc *pairing.Service, version string, ready func() bool) *Server { + if ready == nil { + ready = func() bool { return false } + } + return &Server{pairing: svc, version: version, ready: ready} +} + +// Serve accepts control connections until ctx is cancelled or the listener is +// closed. Clients are served concurrently and each may issue any number of +// sequential requests before disconnecting. +func (s *Server) Serve(ctx context.Context, ln net.Listener) error { + go func() { + <-ctx.Done() + _ = ln.Close() + }() + + var conns sync.WaitGroup + defer conns.Wait() + + for { + conn, err := ln.Accept() + if err != nil { + if ctx.Err() != nil { + return nil + } + // A listener that has been closed is a normal stop; anything else + // is fatal for this endpoint, because Accept will not recover. + if errors.Is(err, net.ErrClosed) { + return nil + } + return fmt.Errorf("accept on the control socket: %w", err) + } + conns.Add(1) + go func() { + defer conns.Done() + defer func() { _ = conn.Close() }() + // A client that is connected but idle must not hold the endpoint + // open past a shutdown: closing its connection unblocks the read + // this goroutine is parked on, so quitting the TUI is prompt even + // with a subcommand still attached. + finished := make(chan struct{}) + defer close(finished) + go func() { + select { + case <-ctx.Done(): + _ = conn.Close() + case <-finished: + } + }() + s.serveConn(ctx, conn) + }() + } +} + +// serveConn reads requests from one client until it disconnects. +func (s *Server) serveConn(ctx context.Context, conn net.Conn) { + codec := jsonrpc.NewCodec(conn) + for { + msg, err := codec.Read() + if err != nil { + var decodeErr *jsonrpc.DecodeError + if errors.As(err, &decodeErr) { + // A malformed line is the client's problem, not the + // endpoint's: say so and keep the connection. + _ = codec.RespondError(nil, CodeParseError, decodeErr.Error()) + continue + } + if !errors.Is(err, io.EOF) { + slog.Debug("control connection ended", "err", err) + } + return + } + if !msg.IsRequest() { + // The control endpoint is request/response only; a notification + // has nobody to answer and is dropped. + continue + } + result, rpcErr := s.dispatch(ctx, msg.Method, msg.Params) + if rpcErr != nil { + if err := codec.RespondErrorData(msg.ID, rpcErr.Code, rpcErr.Message, rpcErr.data); err != nil { + return + } + continue + } + if err := codec.Respond(msg.ID, result); err != nil { + return + } + } +} + +// methodError is a JSON-RPC error a method chose to return. +type methodError struct { + Code int + Message string + data any +} + +func (e *methodError) Error() string { return e.Message } + +func errorf(code int, format string, args ...any) *methodError { + return &methodError{Code: code, Message: fmt.Sprintf(format, args...)} +} + +// dispatch routes one request. The result is returned as json.RawMessage +// wherever the answer is the cluster manager's own Invite, so a caller sees +// exactly what the manager reported rather than a re-encoding of it. +func (s *Server) dispatch(ctx context.Context, method string, params json.RawMessage) (any, *methodError) { + ctx, cancel := context.WithTimeout(ctx, callTimeout) + defer cancel() + + switch method { + case "ping": + return map[string]any{"version": s.version, "brokerReady": s.ready()}, nil + + case "pair:invite": + var p struct { + Address string `json:"address"` + Port int `json:"port"` + NodeID string `json:"nodeId"` + } + if err := decodeParams(params, &p); err != nil { + return nil, errorf(CodeInvalidParams, "%s", err) + } + res, err := s.pairing.Invite(ctx, pairing.InviteRequest{Address: p.Address, Port: p.Port, NodeID: p.NodeID}) + if err != nil { + return nil, relayed(err) + } + return res.Raw, nil + + case "pair:invite-status": + var p struct { + InviteID string `json:"inviteId"` + } + if err := decodeParams(params, &p); err != nil { + return nil, errorf(CodeInvalidParams, "%s", err) + } + if p.InviteID == "" { + return nil, errorf(CodeInvalidParams, "inviteId is required") + } + res, err := s.pairing.InviteStatus(ctx, p.InviteID) + if err != nil { + return nil, relayed(err) + } + return res.Raw, nil + + case "pair:pending": + invites := s.pairing.PendingRaw() + if invites == nil { + invites = []json.RawMessage{} + } + return map[string]any{"invites": invites}, nil + + case "pair:respond": + var p struct { + InviteID string `json:"inviteId"` + Accept *bool `json:"accept"` + Pin string `json:"pin"` + } + if err := decodeParams(params, &p); err != nil { + return nil, errorf(CodeInvalidParams, "%s", err) + } + if p.Accept == nil { + return nil, errorf(CodeInvalidParams, "accept is required") + } + res, err := s.pairing.Respond(ctx, p.InviteID, *p.Accept, p.Pin) + if err != nil { + return nil, relayed(err) + } + return res.Raw, nil + + case "pair:members": + members, err := s.pairing.Members(ctx) + if err != nil { + return nil, relayed(err) + } + return members, nil + + default: + return nil, errorf(CodeMethodNotFound, "unknown method %q", method) + } +} + +// relayed turns a service error into the JSON-RPC error the client sees. A +// cluster-manager error keeps its own code and message, so a caller can key on +// the manager's contract through this endpoint exactly as through the broker. +func relayed(err error) *methodError { + var rpcErr *rpc.RPCError + if errors.As(err, &rpcErr) { + return &methodError{Code: rpcErr.Code, Message: rpcErr.Message} + } + if errors.Is(err, pairing.ErrNoPendingInvite) { + return &methodError{Code: CodeNoPendingInvite, Message: err.Error()} + } + var ambiguous *pairing.AmbiguousInviteError + if errors.As(err, &ambiguous) { + listed := make([]map[string]string, 0, len(ambiguous.Invites)) + for _, inv := range ambiguous.Invites { + listed = append(listed, map[string]string{ + "inviteId": inv.InviteID, + "fromNodeName": inv.FromNodeName, + }) + } + return &methodError{ + Code: CodeAmbiguousInvite, + Message: err.Error(), + data: map[string]any{"invites": listed}, + } + } + return &methodError{Code: CodeInternal, Message: err.Error()} +} + +func decodeParams(raw json.RawMessage, v any) error { + if len(raw) == 0 { + return nil + } + if err := json.Unmarshal(raw, v); err != nil { + return fmt.Errorf("invalid params: %w", err) + } + return nil +} diff --git a/services/nvpair-tui/control/server_test.go b/services/nvpair-tui/control/server_test.go new file mode 100644 index 00000000..d08dc2c0 --- /dev/null +++ b/services/nvpair-tui/control/server_test.go @@ -0,0 +1,417 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package control + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strings" + "sync" + "testing" + "time" + + "nvpair-tui/pairing" + "nvpair-tui/rpc" +) + +// stubBroker answers the cluster-manager methods the endpoint relays. +type stubBroker struct { + mu sync.Mutex + answers map[string]json.RawMessage + errs map[string]*rpc.RPCError + seen []string +} + +func newStubBroker() *stubBroker { + return &stubBroker{answers: map[string]json.RawMessage{}, errs: map[string]*rpc.RPCError{}} +} + +func (s *stubBroker) reply(method, result string) { + s.mu.Lock() + defer s.mu.Unlock() + s.answers[method] = json.RawMessage(result) +} + +func (s *stubBroker) failWith(method string, code int, message string) { + s.mu.Lock() + defer s.mu.Unlock() + s.errs[method] = &rpc.RPCError{Code: code, Message: message} +} + +func (s *stubBroker) Call(_ context.Context, method string, _ any) (*rpc.Message, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.seen = append(s.seen, method) + if err, ok := s.errs[method]; ok { + return nil, err + } + result, ok := s.answers[method] + if !ok { + return nil, fmt.Errorf("stub broker has no answer for %q", method) + } + return &rpc.Message{JSONRPC: "2.0", Result: result}, nil +} + +// endpoint is a running control socket with a client attached, the way a +// subcommand meets one. +type endpoint struct { + client *Client + path string + pairs *pairing.Service + broker *stubBroker +} + +func startEndpoint(t *testing.T, ready bool) *endpoint { + t.Helper() + broker := newStubBroker() + pairs := pairing.NewService(broker) + + path := socketPath(t) + ln, err := Listen(path) + if err != nil { + t.Fatalf("Listen: %v", err) + } + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + defer close(done) + _ = NewServer(pairs, "9.9.9", func() bool { return ready }).Serve(ctx, ln) + }() + + client, err := Dial(path) + if err != nil { + cancel() + t.Fatalf("Dial: %v", err) + } + ep := &endpoint{client: client, path: path, pairs: pairs, broker: broker} + t.Cleanup(func() { + _ = ep.client.Close() + cancel() + _ = ln.Close() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Error("the control server did not stop") + } + }) + return ep +} + +func (e *endpoint) call(t *testing.T, method string, params any) json.RawMessage { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + raw, err := e.client.Call(ctx, method, params) + if err != nil { + t.Fatalf("%s: %v", method, err) + } + return raw +} + +func (e *endpoint) callErr(t *testing.T, method string, params any) *Error { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _, err := e.client.Call(ctx, method, params) + if err == nil { + t.Fatalf("%s succeeded; an error was expected", method) + } + rpcErr, ok := err.(*Error) + if !ok { + t.Fatalf("%s returned %T (%v), want a JSON-RPC error", method, err, err) + } + return rpcErr +} + +func TestPingReportsVersionAndBrokerReadiness(t *testing.T) { + e := startEndpoint(t, true) + var got struct { + Version string `json:"version"` + BrokerReady bool `json:"brokerReady"` + } + if err := json.Unmarshal(e.call(t, "ping", nil), &got); err != nil { + t.Fatalf("decode ping: %v", err) + } + if got.Version != "9.9.9" || !got.BrokerReady { + t.Errorf("ping = %+v, want the stamped version and a ready broker", got) + } +} + +func TestPingReportsABrokerThatHasNotAnnouncedItself(t *testing.T) { + e := startEndpoint(t, false) + var got struct { + BrokerReady bool `json:"brokerReady"` + } + if err := json.Unmarshal(e.call(t, "ping", nil), &got); err != nil { + t.Fatalf("decode ping: %v", err) + } + if got.BrokerReady { + t.Error("brokerReady = true before the broker announced itself") + } +} + +func TestPairInviteRelaysTheManagersOwnInvite(t *testing.T) { + e := startEndpoint(t, true) + e.broker.reply("cluster:invite-node", + `{"inviteId":"inv-1","state":"pending","pin":"402199","clusterFriendlyName":"Lab 3 desks","somethingNew":42}`) + + raw := e.call(t, "pair:invite", map[string]any{"address": "10.0.0.5", "port": 14321}) + var fields map[string]any + if err := json.Unmarshal(raw, &fields); err != nil { + t.Fatalf("decode: %v", err) + } + if fields["pin"] != "402199" { + t.Errorf("pin = %v, want the manager's own", fields["pin"]) + } + // A field this package does not model must survive the relay, or the + // endpoint would silently drop parts of the manager's contract. + if fields["somethingNew"] != float64(42) { + t.Errorf("the relay dropped an unmodelled field: %v", fields) + } +} + +func TestPairInviteRejectsAnEmptyAddress(t *testing.T) { + e := startEndpoint(t, true) + rpcErr := e.callErr(t, "pair:invite", map[string]any{"address": ""}) + if !strings.Contains(rpcErr.Message, "address is required") { + t.Errorf("message = %q, want it to name the missing address", rpcErr.Message) + } +} + +func TestPairInviteRelaysAManagerError(t *testing.T) { + e := startEndpoint(t, true) + e.broker.failWith("cluster:invite-node", -32603, "cluster manager is not available") + + rpcErr := e.callErr(t, "pair:invite", map[string]any{"address": "10.0.0.5"}) + if rpcErr.Code != -32603 { + t.Errorf("code = %d, want the manager's own code relayed", rpcErr.Code) + } + if rpcErr.Message != "cluster manager is not available" { + t.Errorf("message = %q, want the manager's own", rpcErr.Message) + } +} + +func TestPairInviteStatusRequiresAnInviteID(t *testing.T) { + e := startEndpoint(t, true) + rpcErr := e.callErr(t, "pair:invite-status", map[string]any{}) + if rpcErr.Code != CodeInvalidParams { + t.Errorf("code = %d, want %d", rpcErr.Code, CodeInvalidParams) + } +} + +func TestPairInviteStatusReturnsTheCurrentInvite(t *testing.T) { + e := startEndpoint(t, true) + e.broker.reply("cluster:invite-status", `{"inviteId":"inv-1","state":"paired"}`) + + var got pairing.Invite + if err := json.Unmarshal(e.call(t, "pair:invite-status", map[string]any{"inviteId": "inv-1"}), &got); err != nil { + t.Fatalf("decode: %v", err) + } + if got.State != pairing.StatePaired { + t.Errorf("state = %q, want paired", got.State) + } +} + +func TestPairPendingIsAnEmptyListWhenNothingIsWaiting(t *testing.T) { + e := startEndpoint(t, true) + raw := e.call(t, "pair:pending", nil) + if string(raw) != `{"invites":[]}` { + t.Errorf("pair:pending = %s, want an empty list rather than null", raw) + } +} + +func TestPairPendingListsInboundInvitesWithoutAPin(t *testing.T) { + e := startEndpoint(t, true) + e.pairs.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(`{"inviteId":"inv-1","fromNodeName":"Lab desk A","state":"pending","pin":null}`), + }) + + var got struct { + Invites []map[string]any `json:"invites"` + } + if err := json.Unmarshal(e.call(t, "pair:pending", nil), &got); err != nil { + t.Fatalf("decode: %v", err) + } + if len(got.Invites) != 1 { + t.Fatalf("pending = %d invites, want 1", len(got.Invites)) + } + if got.Invites[0]["inviteId"] != "inv-1" { + t.Errorf("inviteId = %v", got.Invites[0]["inviteId"]) + } + if _, ok := got.Invites[0]["pin"]; ok { + t.Error("pair:pending disclosed a pin member") + } + if got.Invites[0]["receivedAt"] == nil { + t.Error("pair:pending did not stamp receivedAt, so a caller cannot age the invite") + } +} + +func TestPairRespondRequiresAcceptToBeStated(t *testing.T) { + e := startEndpoint(t, true) + rpcErr := e.callErr(t, "pair:respond", map[string]any{"pin": "402199"}) + if rpcErr.Code != CodeInvalidParams || !strings.Contains(rpcErr.Message, "accept is required") { + t.Errorf("error = %+v, want accept to be required", rpcErr) + } +} + +func TestPairRespondWithNothingPending(t *testing.T) { + e := startEndpoint(t, true) + rpcErr := e.callErr(t, "pair:respond", map[string]any{"accept": true, "pin": "402199"}) + if rpcErr.Code != CodeNoPendingInvite { + t.Errorf("code = %d, want %d", rpcErr.Code, CodeNoPendingInvite) + } +} + +func TestPairRespondWithSeveralPendingNamesThem(t *testing.T) { + e := startEndpoint(t, true) + for _, id := range []string{"inv-1", "inv-2"} { + e.pairs.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(fmt.Sprintf(`{"inviteId":%q,"fromNodeName":"Lab desk %s","state":"pending"}`, id, id)), + }) + } + + rpcErr := e.callErr(t, "pair:respond", map[string]any{"accept": true, "pin": "402199"}) + if rpcErr.Code != CodeAmbiguousInvite { + t.Fatalf("code = %d, want %d", rpcErr.Code, CodeAmbiguousInvite) + } + var data struct { + Invites []struct { + InviteID string `json:"inviteId"` + FromNodeName string `json:"fromNodeName"` + } `json:"invites"` + } + if err := json.Unmarshal(rpcErr.Data, &data); err != nil { + t.Fatalf("decode error data: %v", err) + } + if len(data.Invites) != 2 { + t.Fatalf("error data named %d invites, want both", len(data.Invites)) + } + for _, want := range []string{"inv-1", "inv-2"} { + if !strings.Contains(rpcErr.Message, want) { + t.Errorf("message %q does not name %q", rpcErr.Message, want) + } + } +} + +func TestPairRespondAcceptsTheSolePendingInvite(t *testing.T) { + e := startEndpoint(t, true) + e.pairs.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(`{"inviteId":"inv-1","fromNodeName":"Lab desk A","state":"pending"}`), + }) + e.broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-1","state":"paired"}`) + + var got pairing.Invite + if err := json.Unmarshal(e.call(t, "pair:respond", map[string]any{"accept": true, "pin": "402199"}), &got); err != nil { + t.Fatalf("decode: %v", err) + } + if got.State != pairing.StatePaired { + t.Errorf("state = %q, want paired", got.State) + } + raw := e.call(t, "pair:pending", nil) + if string(raw) != `{"invites":[]}` { + t.Errorf("the answered invite is still pending: %s", raw) + } +} + +func TestPairRespondReportsAWrongPinAsAResult(t *testing.T) { + e := startEndpoint(t, true) + e.pairs.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(`{"inviteId":"inv-1","state":"pending"}`), + }) + e.broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-1","state":"failed","reason":"incorrect-pin"}`) + + var got pairing.Invite + if err := json.Unmarshal(e.call(t, "pair:respond", map[string]any{"accept": true, "pin": "000000"}), &got); err != nil { + t.Fatalf("decode: %v", err) + } + if got.State != pairing.StateFailed || got.Reason != pairing.ReasonIncorrectPin { + t.Errorf("invite = %+v, want a failed result carrying incorrect-pin", got) + } +} + +func TestPairMembers(t *testing.T) { + e := startEndpoint(t, true) + e.broker.reply("cluster:get-node-id", + `{"nodeUuid":"uuid-a","nodeId":"NODE-A","name":"Lab desk A","clusterId":"cluster-xyz","clusterFriendlyName":"Lab 3 desks"}`) + e.broker.reply("nodes:get-initial", + `{"nodes":[{"id":"NODE-B","nodeUuid":"uuid-b","name":"Lab desk B","ipAddress":"10.0.0.5","port":14321,"state":"member"}]}`) + + var got pairing.Membership + if err := json.Unmarshal(e.call(t, "pair:members", nil), &got); err != nil { + t.Fatalf("decode: %v", err) + } + if got.ClusterID != "cluster-xyz" || got.NodeID != "NODE-A" { + t.Errorf("identity = %+v", got) + } + if len(got.Members) != 1 || got.Members[0].ID != "NODE-B" { + t.Errorf("members = %+v", got.Members) + } +} + +func TestUnknownMethod(t *testing.T) { + e := startEndpoint(t, true) + rpcErr := e.callErr(t, "pair:teleport", nil) + if rpcErr.Code != CodeMethodNotFound { + t.Errorf("code = %d, want %d", rpcErr.Code, CodeMethodNotFound) + } +} + +func TestTheEndpointServesSequentialClients(t *testing.T) { + e := startEndpoint(t, true) + // The first client is the one startEndpoint opened; use it, close it, and + // then reach the same endpoint again — the shape every subcommand has, + // each being its own short-lived process. + e.call(t, "ping", nil) + if err := e.client.Close(); err != nil { + t.Fatalf("close the first client: %v", err) + } + + for i := 0; i < 3; i++ { + client, err := Dial(e.path) + if err != nil { + t.Fatalf("client %d could not reach the endpoint: %v", i, err) + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + if _, err := client.Call(ctx, "ping", nil); err != nil { + cancel() + _ = client.Close() + t.Fatalf("client %d ping: %v", i, err) + } + cancel() + _ = client.Close() + } + // Keep the cleanup's Close idempotent-safe by handing back a live client. + client, err := Dial(e.path) + if err != nil { + t.Fatalf("reopen: %v", err) + } + e.client = client +} + +func TestDialReportsNoRunningInstance(t *testing.T) { + _, err := Dial(socketPath(t)) + if err == nil { + t.Fatal("Dial succeeded against an endpoint that does not exist") + } + if !errors.Is(err, ErrNotRunning) { + t.Fatalf("err = %T (%v), want it to unwrap to ErrNotRunning", err, err) + } + var notRunning *NotRunningError + if !errors.As(err, ¬Running) { + t.Fatalf("err = %T (%v), want *NotRunningError", err, err) + } + if !strings.Contains(err.Error(), "no nvpair-tui is listening") { + t.Errorf("message = %q, want it to say nothing is listening", err) + } +} diff --git a/services/nvpair-tui/go.mod b/services/nvpair-tui/go.mod index 486f0978..ff360064 100644 --- a/services/nvpair-tui/go.mod +++ b/services/nvpair-tui/go.mod @@ -10,6 +10,7 @@ require ( ) require ( + github.com/Microsoft/go-winio v0.6.2 // indirect github.com/atotto/clipboard v0.1.4 // indirect github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect github.com/charmbracelet/colorprofile v0.4.1 // indirect diff --git a/services/nvpair-tui/go.sum b/services/nvpair-tui/go.sum index 72298701..ddda2770 100644 --- a/services/nvpair-tui/go.sum +++ b/services/nvpair-tui/go.sum @@ -1,3 +1,5 @@ +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4= github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI= github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k= diff --git a/services/nvpair-tui/main.go b/services/nvpair-tui/main.go index 4ee976f9..eb93e210 100644 --- a/services/nvpair-tui/main.go +++ b/services/nvpair-tui/main.go @@ -6,9 +6,14 @@ // connection. It is designed to run comfortably over SSH on a headless // server where the bundled graphical UI cannot run. // +// Given a subcommand instead of flags it does the opposite: it starts nothing +// and connects to the control socket of an nvpair-tui already running on this +// machine, so pairing can be driven from a script. See cli.go. +// // This file is the process entrypoint: it parses flags, initialises -// logging, spawns the broker, and drives the supervisor. Logging goes to -// stderr so it never collides with the full-screen TUI on stdout. +// logging, spawns the broker, serves the control socket, and drives the +// supervisor. Logging goes to stderr so it never collides with the +// full-screen TUI on stdout. package main import ( @@ -18,9 +23,13 @@ import ( "log/slog" "os" "os/signal" + "sync/atomic" "syscall" "nvpair-shared/applog" + "nvpair-tui/control" + "nvpair-tui/pairing" + "nvpair-tui/rpc" "nvpair-tui/ui" ) @@ -30,9 +39,22 @@ import ( var Version = "dev" func main() { + // A subcommand drives an already-running instance and must not start a + // broker, a UI, or a second control socket of its own. + if name := subcommandName(os.Args[1:]); name != "" { + os.Exit(runSubcommand(newCLIEnv(os.Stdout, os.Stderr), os.Args[1:])) + } + brokerPath := flag.String("broker-path", "", "path to nvpair-ui-broker binary (default: ./nvpair-ui-broker alongside this executable)") + controlSocket := flag.String("control-socket", "", "path to the local control socket to serve (default: the per-user path)") + noControlSocket := flag.Bool("no-control-socket", false, "do not serve the local control socket (disables the nvpair-tui subcommands against this instance)") showVersion := flag.Bool("version", false, "print version and exit") resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo) + flag.Usage = func() { + printUsage(flag.CommandLine.Output()) + fmt.Fprintln(flag.CommandLine.Output(), "\nFlags:") + flag.PrintDefaults() + } flag.Parse() if *showVersion { @@ -67,13 +89,106 @@ func main() { os.Exit(1) } + // One pairing service drives both the Cluster tab and the control socket, + // so the two can never hold different ideas of what is pending. + pairs := pairing.NewService(sup.Client) + + // The broker pushes on a single channel with a single consumer, so the + // fan-out happens here rather than inside the UI: the pairing service has + // to see cluster:invite-received whether or not the UI is keeping up. + var brokerReady atomic.Bool + uiNotifications := make(chan *rpc.Message, notificationBuffer) + go fanOutNotifications(sup.Client.Notifications(), pairs, &brokerReady, uiNotifications) + + stopControl := serveControlSocket(ctx, *controlSocket, *noControlSocket, pairs, &brokerReady) + // The broker's stderr (its logs plus every worker's, prefixed) is fed // into the UI's Logs view rather than the terminal, so it never // collides with the full-screen TUI on stdout. - if err := ui.Run(sup.Client, sup.Stderr); err != nil { + if err := ui.Run(ui.Deps{ + Client: sup.Client, + Notifications: uiNotifications, + Stderr: sup.Stderr, + Pairing: pairs, + }); err != nil { slog.Error("ui error", "err", err) } + stopControl() sup.Shutdown() slog.Info("shutdown complete") } + +// notificationBuffer bounds the queue of broker pushes waiting for the UI. It +// matches the client's own so the fan-out adds no new stall point. +const notificationBuffer = 256 + +// fanOutNotifications delivers every broker push to the pairing service and +// then to the UI, and records the broker's readiness on the way past. +// +// The pairing service is fed first and synchronously: it must observe an +// invite even if the UI is between frames. The UI's copy is dropped rather +// than blocked on, because a stalled UI must not stall pairing. +// +// The stream closing means the broker is gone, so readiness is withdrawn on +// the way out: a script that asks `ping` after the broker died has to be told +// the truth, or it will go on to send an invite that can only time out. +func fanOutNotifications(in <-chan *rpc.Message, pairs *pairing.Service, ready *atomic.Bool, out chan<- *rpc.Message) { + defer close(out) + defer ready.Store(false) + for msg := range in { + if msg.Method == "app:ready" { + ready.Store(true) + } + pairs.HandleNotification(msg) + select { + case out <- msg: + default: + slog.Debug("dropped a broker notification for the UI", "method", msg.Method) + } + } +} + +// serveControlSocket opens the local control endpoint and starts serving it, +// returning a function that stops it. +// +// A control socket that cannot be opened is a warning, not a failure: the +// interactive UI is nvpair-tui's primary job and it runs without one. The +// warning names the endpoint so an operator whose subcommands fail can see +// why. +func serveControlSocket(ctx context.Context, override string, disabled bool, pairs *pairing.Service, ready *atomic.Bool) func() { + if disabled { + slog.Info("control socket disabled by --no-control-socket") + return func() {} + } + path := override + if path == "" { + resolved, err := control.DefaultPath() + if err != nil { + slog.Warn("no control socket: could not resolve its path", "err", err) + return func() {} + } + path = resolved + } + listener, err := control.Listen(path) + if err != nil { + slog.Warn("no control socket: the nvpair-tui subcommands will not reach this instance", + "path", path, "err", err) + return func() {} + } + slog.Info("control socket listening", "path", path) + + serveCtx, stop := context.WithCancel(ctx) + done := make(chan struct{}) + go func() { + defer close(done) + if err := control.NewServer(pairs, Version, ready.Load).Serve(serveCtx, listener); err != nil { + slog.Warn("control socket stopped", "err", err) + } + }() + return func() { + stop() + _ = listener.Close() + <-done + } +} diff --git a/services/nvpair-tui/notifications_test.go b/services/nvpair-tui/notifications_test.go new file mode 100644 index 00000000..aeea0d33 --- /dev/null +++ b/services/nvpair-tui/notifications_test.go @@ -0,0 +1,89 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "sync/atomic" + "testing" + "time" + + "nvpair-tui/pairing" + "nvpair-tui/rpc" +) + +// TestTheFanOutFeedsPairingBeforeTheUI checks the property the whole design +// rests on: the pairing service observes an invitation whether or not the UI is +// reading. The UI channel here is left unread and undersized on purpose. +func TestTheFanOutFeedsPairingBeforeTheUI(t *testing.T) { + pushes := make(chan *rpc.Message, 1) + toUI := make(chan *rpc.Message) // nobody reads this + pairs := pairing.NewService(nil) + var ready atomic.Bool + + done := make(chan struct{}) + go func() { + defer close(done) + fanOutNotifications(pushes, pairs, &ready, toUI) + }() + + pushes <- &rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(`{"inviteId":"inv-1","fromNodeName":"Lab desk A","state":"pending"}`), + } + deadline := time.Now().Add(5 * time.Second) + for len(pairs.Pending()) == 0 { + if time.Now().After(deadline) { + t.Fatal("the pairing service never saw the invitation, because the UI was not reading") + } + time.Sleep(5 * time.Millisecond) + } + + close(pushes) + <-done +} + +// TestReadinessIsWithdrawnWhenTheBrokerGoesAway guards the answer `ping` gives +// a script after the broker has died. Reporting a ready broker there would send +// the script on to an invite that can only time out. +func TestReadinessIsWithdrawnWhenTheBrokerGoesAway(t *testing.T) { + pushes := make(chan *rpc.Message, 2) + toUI := make(chan *rpc.Message, 8) + var ready atomic.Bool + + done := make(chan struct{}) + go func() { + defer close(done) + fanOutNotifications(pushes, pairing.NewService(nil), &ready, toUI) + }() + + pushes <- &rpc.Message{JSONRPC: "2.0", Method: "app:ready", Params: json.RawMessage(`{"version":"1.2.3"}`)} + deadline := time.Now().Add(5 * time.Second) + for !ready.Load() { + if time.Now().After(deadline) { + t.Fatal("app:ready did not mark the broker ready") + } + time.Sleep(5 * time.Millisecond) + } + + // The broker's notification stream closing is how this process learns the + // broker is gone. + close(pushes) + <-done + + if ready.Load() { + t.Error("the broker is gone but ping would still report it ready") + } + // app:ready was forwarded to the UI as well, so drain what is buffered and + // then confirm the channel is closed rather than merely empty — that close + // is what makes the UI render "broker disconnected". + forwarded := 0 + for range toUI { + forwarded++ + } + if forwarded != 1 { + t.Errorf("the UI received %d notifications, want the one app:ready", forwarded) + } +} diff --git a/services/nvpair-tui/pairing/invite.go b/services/nvpair-tui/pairing/invite.go new file mode 100644 index 00000000..30340639 --- /dev/null +++ b/services/nvpair-tui/pairing/invite.go @@ -0,0 +1,209 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Package pairing is the single implementation of node pairing inside +// nvpair-tui. Both drivers — the interactive Cluster tab and the control +// socket that backs the non-interactive subcommands — go through one +// Service, so the two can never disagree about what an invite is or which +// one is pending. +// +// Everything here is expressed in the vocabulary nvpair-cluster-manager +// already uses (see ../../nvpair-cluster-manager/README.md); this package +// adds no pairing semantics of its own. It only relays through the broker, +// remembers which inbound invites are still unanswered, and tells its +// subscribers when that changed. +package pairing + +import ( + "encoding/json" + "errors" + "fmt" + "net" + "strconv" + "strings" +) + +// Invite mirrors nvpair-cluster-manager's Invite: one pairing session, from +// the invite-node result, an invite-status poll, or an invite-* push. +// +// Pin is populated only in the inviting node's own invite-node result. It is +// a secret carried by a human out of band: it may be printed to the terminal +// the operator asked for and returned over the control socket, and it must +// never reach a log. Nothing in this package logs an Invite. +type Invite struct { + InviteID string `json:"inviteId"` + FromNodeID string `json:"fromNodeId,omitempty"` + FromNodeUUID string `json:"fromNodeUuid,omitempty"` + FromNodeName string `json:"fromNodeName,omitempty"` + ToNodeID *string `json:"toNodeId,omitempty"` + ClusterID string `json:"clusterId,omitempty"` + ClusterFriendlyName string `json:"clusterFriendlyName,omitempty"` + Pin *string `json:"pin,omitempty"` + State string `json:"state"` + Reason string `json:"reason,omitempty"` + CreatedAt int64 `json:"createdAt,omitempty"` + RespondedAt *int64 `json:"respondedAt,omitempty"` + + // ReceivedAt is this node's own wall clock (epoch ms) at the moment the + // invite-received push arrived. CreatedAt is the *inviter's* clock, which + // is why an age computed from it can come out negative between two + // unsynchronised machines. Set only on inbound invites. + ReceivedAt int64 `json:"receivedAt,omitempty"` +} + +// Invite states, as nvpair-cluster-manager reports them. +const ( + StatePending = "pending" + StatePaired = "paired" + StateDeclined = "declined" + StateCanceled = "canceled" + StateExpired = "expired" + StateFailed = "failed" + StateRejected = "rejected" +) + +// ReasonIncorrectPin is the failure reason for a well-formed but wrong PIN. +// A malformed PIN is a JSON-RPC error instead, not a failed result. +const ReasonIncorrectPin = "incorrect-pin" + +// Terminal reports whether the invite has reached a state it will not leave, +// so a caller waiting on it can stop. +func (i Invite) Terminal() bool { + switch i.State { + case "", StatePending: + return false + default: + return true + } +} + +// PIN returns the six-digit pairing PIN, or "" when this invite carries none +// (every invite but the inviter's own invite-node result). +func (i Invite) PIN() string { + if i.Pin == nil { + return "" + } + return *i.Pin +} + +// Describe names the invite for a human without disclosing the PIN. It is +// safe to put in an error message that may be logged. +func (i Invite) Describe() string { + if i.FromNodeName == "" { + return i.InviteID + } + return fmt.Sprintf("%s (from %s)", i.InviteID, i.FromNodeName) +} + +// Result is one broker Invite response: the manager's own JSON, relayed +// untouched so the control socket hands callers exactly what the manager +// said, plus the decode this package and the CLI act on. +type Result struct { + Raw json.RawMessage + Invite Invite +} + +// ErrNoPendingInvite is returned when a response was asked for without an +// invite id and no inbound invite is waiting. +var ErrNoPendingInvite = errors.New("no invite is pending on this node") + +// AmbiguousInviteError is returned when a response was asked for without an +// invite id and more than one inbound invite is waiting. It names them so the +// operator can pick one with --invite. +type AmbiguousInviteError struct{ Invites []Invite } + +func (e *AmbiguousInviteError) Error() string { + names := make([]string, 0, len(e.Invites)) + for _, inv := range e.Invites { + names = append(names, inv.Describe()) + } + return fmt.Sprintf("%d invites are pending; name one with --invite: %s", + len(e.Invites), strings.Join(names, ", ")) +} + +// InviteRequest is an outbound pairing target. Address is a bare host; Port +// overrides the manager's default pairing port; NodeID pins the target's +// identity when the caller already knows it (the Nodes tab does, an operator +// typing an address does not). +type InviteRequest struct { + Address string + Port int + NodeID string +} + +// params renders the request as cluster:invite-node params. +// +// The manager takes "address" as a bare host and appends the port itself +// (default 14321), so a "host:port" the operator typed has to be split: glued +// together it would dial [host:port]:14321. An explicit Port always wins over +// one embedded in Address. +func (r InviteRequest) params() (map[string]any, error) { + address := strings.TrimSpace(r.Address) + if address == "" { + return nil, errors.New("an address is required to invite a node") + } + port := r.Port + if port == 0 { + if host, portStr, err := net.SplitHostPort(address); err == nil { + if p, perr := strconv.Atoi(portStr); perr == nil { + address, port = host, p + } + } + } + if port < 0 || port > 65535 { + return nil, fmt.Errorf("port %d is out of range", port) + } + params := map[string]any{"address": address} + if port > 0 { + params["port"] = port + } + if r.NodeID != "" { + params["nodeId"] = r.NodeID + } + return params, nil +} + +// ClusterNode mirrors nvpair-cluster-manager's ClusterNode: a cluster member +// or a node with a pairing still in flight. +type ClusterNode struct { + ID string `json:"id"` + NodeUUID string `json:"nodeUuid"` + Name string `json:"name"` + IPAddress string `json:"ipAddress"` + Port int `json:"port"` + ClusterID string `json:"clusterId,omitempty"` + State string `json:"state"` + JoinedAt *int64 `json:"joinedAt,omitempty"` + LastSeen *int64 `json:"lastSeen,omitempty"` +} + +// Membership is this node's own cluster identity plus its roster, the answer +// to pair:members. +type Membership struct { + ClusterID string `json:"clusterId"` + ClusterFriendlyName string `json:"clusterFriendlyName,omitempty"` + NodeID string `json:"nodeId"` + NodeUUID string `json:"nodeUuid"` + Name string `json:"name"` + Members []ClusterNode `json:"members"` +} + +// stripPin returns raw with any "pin" member removed and receivedAt stamped +// in, preserving every other field the manager sent (including ones this +// package does not model). A blob that will not parse as an object yields +// nil, and the caller falls back to re-encoding the decoded Invite. +func stripPin(raw json.RawMessage, receivedAt int64) json.RawMessage { + var fields map[string]json.RawMessage + if err := json.Unmarshal(raw, &fields); err != nil { + return nil + } + delete(fields, "pin") + if receivedAt > 0 { + fields["receivedAt"] = json.RawMessage(strconv.FormatInt(receivedAt, 10)) + } + out, err := json.Marshal(fields) + if err != nil { + return nil + } + return out +} diff --git a/services/nvpair-tui/pairing/service.go b/services/nvpair-tui/pairing/service.go new file mode 100644 index 00000000..0232a6e9 --- /dev/null +++ b/services/nvpair-tui/pairing/service.go @@ -0,0 +1,430 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package pairing + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "sort" + "sync" + "time" + + "nvpair-tui/rpc" +) + +// Broker is the half of the broker client this package needs. The TUI's +// rpc.Client satisfies it; a test supplies its own. +type Broker interface { + Call(ctx context.Context, method string, params any) (*rpc.Message, error) +} + +// eventBuffer bounds a subscriber's queue. Pairing events are rare — a +// handful per pairing — so this only has to absorb the case of a subscriber +// that is briefly not reading (the Bubble Tea loop between frames). A +// subscriber that falls this far behind loses events rather than stalling +// the pairing that produced them. +const eventBuffer = 64 + +// EventKind names what happened to a pairing. Subscribers switch on it. +type EventKind string + +const ( + // EventInviteSent — this node created an outbound invite. The event + // carries the PIN, which is why it goes only to in-process subscribers + // (the Cluster tab's status line) and never to a log. + EventInviteSent EventKind = "invite-sent" + // EventInviteRejected — the target refused to pair (already clustered). + EventInviteRejected EventKind = "invite-rejected" + // EventInviteFailed — an outbound invite could not be created at all. + EventInviteFailed EventKind = "invite-failed" + // EventInviteReceived — an inbound invite is now waiting for an answer. + EventInviteReceived EventKind = "invite-received" + // EventInviteCleared — a waiting inbound invite went away on its own + // (canceled by the inviter, superseded, or expired). + EventInviteCleared EventKind = "invite-cleared" + // EventResponded — this node answered an inbound invite. + EventResponded EventKind = "responded" + // EventRespondFailed — answering an inbound invite errored out. + EventRespondFailed EventKind = "respond-failed" +) + +// Event is one pairing state change, delivered to every subscriber. Invite is +// the session it concerns; Accept distinguishes an accept from a decline on +// EventResponded; Err carries the failure on the *Failed kinds. +type Event struct { + Kind EventKind + Invite Invite + Accept bool + Err error +} + +// pendingInvite is an inbound invite still waiting for an answer: the decode +// this package reasons about, and the manager's own JSON with the pin removed +// and receivedAt stamped in, which is what pair:pending returns. +type pendingInvite struct { + invite Invite + raw json.RawMessage +} + +// Service drives pairing over one broker connection and remembers which +// inbound invites are still unanswered. +// +// Its pending set is deliberately session state: nvpair-cluster-manager has no +// "list the invites you are holding" call, and nodes:get-initial reports a +// pending-inbound peer without the invite id needed to answer it. A TUI +// restarted mid-pairing therefore reports nothing pending even though the +// manager still holds a live invite; the inviter has to re-invite. This is +// recorded in the component README. +type Service struct { + broker Broker + now func() time.Time + + mu sync.Mutex + pending map[string]pendingInvite + arrived []string // pending ids, oldest first + subs map[int]chan Event + nextSub int +} + +// NewService builds a Service over a connected broker client. +func NewService(broker Broker) *Service { + return &Service{ + broker: broker, + now: time.Now, + pending: make(map[string]pendingInvite), + subs: make(map[int]chan Event), + } +} + +// Subscribe returns a channel of pairing events and a function that stops the +// subscription and releases the channel. Every subscriber sees every event +// from the moment it subscribes. +func (s *Service) Subscribe() (<-chan Event, func()) { + ch := make(chan Event, eventBuffer) + s.mu.Lock() + id := s.nextSub + s.nextSub++ + s.subs[id] = ch + s.mu.Unlock() + + var once sync.Once + return ch, func() { + once.Do(func() { + s.mu.Lock() + delete(s.subs, id) + s.mu.Unlock() + close(ch) + }) + } +} + +// publish fans an event out to every subscriber. A subscriber whose buffer is +// full loses this event: a pairing must never block on a slow reader. +// +// The caller must not hold s.mu — a subscriber may call back into the Service +// from its own goroutine. +func (s *Service) publish(ev Event) { + s.mu.Lock() + targets := make([]chan Event, 0, len(s.subs)) + for _, ch := range s.subs { + targets = append(targets, ch) + } + s.mu.Unlock() + for _, ch := range targets { + select { + case ch <- ev: + default: + } + } +} + +// Invite asks this node's cluster manager to pair with req's target and +// returns the manager's Invite. On success the result carries the six-digit +// PIN the operator reads to the other machine. +// +// If this node is not in a cluster yet the manager auto-founds a cluster of +// one first; there is no separate create step to drive. +func (s *Service) Invite(ctx context.Context, req InviteRequest) (Result, error) { + params, err := req.params() + if err != nil { + s.publish(Event{Kind: EventInviteFailed, Err: err}) + return Result{}, err + } + res, err := s.call(ctx, "cluster:invite-node", params) + if err != nil { + s.publish(Event{Kind: EventInviteFailed, Err: err}) + return Result{}, err + } + if res.Invite.State == StateRejected { + s.publish(Event{Kind: EventInviteRejected, Invite: res.Invite}) + return res, nil + } + s.publish(Event{Kind: EventInviteSent, Invite: res.Invite}) + return res, nil +} + +// InviteStatus returns the manager's current view of one invite, whichever +// side of the pairing created it. +func (s *Service) InviteStatus(ctx context.Context, inviteID string) (Result, error) { + if inviteID == "" { + return Result{}, fmt.Errorf("an invite id is required") + } + return s.call(ctx, "cluster:invite-status", map[string]any{"inviteId": inviteID}) +} + +// Pending returns the inbound invites still waiting for an answer, oldest +// first. The PIN is never part of an inbound invite, and is stripped again +// here so no caller can leak one it did not have. +func (s *Service) Pending() []Invite { + s.mu.Lock() + defer s.mu.Unlock() + out := make([]Invite, 0, len(s.arrived)) + for _, id := range s.arrived { + if p, ok := s.pending[id]; ok { + inv := p.invite + inv.Pin = nil + out = append(out, inv) + } + } + return out +} + +// PendingRaw returns the same invites as Pending, each as the manager's own +// JSON with the pin removed and receivedAt stamped in. The control socket +// returns these so a caller sees exactly what the manager reported. +func (s *Service) PendingRaw() []json.RawMessage { + s.mu.Lock() + defer s.mu.Unlock() + out := make([]json.RawMessage, 0, len(s.arrived)) + for _, id := range s.arrived { + p, ok := s.pending[id] + if !ok { + continue + } + if p.raw != nil { + out = append(out, p.raw) + continue + } + inv := p.invite + inv.Pin = nil + if encoded, err := json.Marshal(inv); err == nil { + out = append(out, encoded) + } + } + return out +} + +// Respond answers an inbound invite. An empty inviteID means "the one that is +// pending": with none waiting that is ErrNoPendingInvite, and with several it +// is an *AmbiguousInviteError naming them all. +// +// A wrong PIN is not an error here — it comes back as a successful result in +// state "failed" with reason "incorrect-pin", which is what the manager +// reports and what the caller's exit code turns on. +func (s *Service) Respond(ctx context.Context, inviteID string, accept bool, pin string) (Result, error) { + if inviteID == "" { + resolved, err := s.solePending() + if err != nil { + s.publish(Event{Kind: EventRespondFailed, Accept: accept, Err: err}) + return Result{}, err + } + inviteID = resolved + } + params := map[string]any{"inviteId": inviteID, "accept": accept} + if accept && pin != "" { + params["pin"] = pin + } + res, err := s.call(ctx, "cluster:respond-to-invite", params) + if err != nil { + // An invite the manager does not know, or one it considers already + // terminal, will never be answerable. Drop it so it cannot sit in + // pair:pending forever. + if unanswerable(err) { + s.forget(inviteID) + } + s.publish(Event{Kind: EventRespondFailed, Invite: Invite{InviteID: inviteID}, Accept: accept, Err: err}) + return Result{}, err + } + s.forget(inviteID) + s.publish(Event{Kind: EventResponded, Invite: res.Invite, Accept: accept}) + return res, nil +} + +// Members returns this node's cluster identity and roster. +func (s *Service) Members(ctx context.Context) (Membership, error) { + idMsg, err := s.broker.Call(ctx, "cluster:get-node-id", nil) + if err != nil { + return Membership{}, err + } + var identity struct { + NodeUUID string `json:"nodeUuid"` + NodeID string `json:"nodeId"` + Name string `json:"name"` + ClusterID string `json:"clusterId"` + ClusterFriendlyName string `json:"clusterFriendlyName"` + } + if err := decode(idMsg.Result, &identity); err != nil { + return Membership{}, fmt.Errorf("decode cluster:get-node-id: %w", err) + } + + nodesMsg, err := s.broker.Call(ctx, "nodes:get-initial", nil) + if err != nil { + return Membership{}, err + } + var roster struct { + Nodes []ClusterNode `json:"nodes"` + } + if err := decode(nodesMsg.Result, &roster); err != nil { + return Membership{}, fmt.Errorf("decode nodes:get-initial: %w", err) + } + if roster.Nodes == nil { + roster.Nodes = []ClusterNode{} + } + return Membership{ + ClusterID: identity.ClusterID, + ClusterFriendlyName: identity.ClusterFriendlyName, + NodeID: identity.NodeID, + NodeUUID: identity.NodeUUID, + Name: identity.Name, + Members: roster.Nodes, + }, nil +} + +// AwaitPending returns an inbound invite, immediately if one is already +// waiting and otherwise as soon as one arrives. It is what `accept --wait` +// blocks on. ctx bounds the wait. +func (s *Service) AwaitPending(ctx context.Context) (Invite, error) { + // Subscribe before looking, so an invite that lands between the two is + // seen on the channel rather than missed by both. + events, unsubscribe := s.Subscribe() + defer unsubscribe() + + if waiting := s.Pending(); len(waiting) > 0 { + return waiting[0], nil + } + for { + select { + case <-ctx.Done(): + return Invite{}, ctx.Err() + case ev, ok := <-events: + if !ok { + return Invite{}, fmt.Errorf("pairing service stopped while waiting for an invite") + } + if ev.Kind == EventInviteReceived { + return ev.Invite, nil + } + } + } +} + +// HandleNotification folds one broker push into the pending set. The caller +// feeds it every notification; anything that is not a pairing push is ignored. +func (s *Service) HandleNotification(msg *rpc.Message) { + if msg == nil { + return + } + switch msg.Method { + case "cluster:invite-received": + var inv Invite + if err := decode(msg.Params, &inv); err != nil || inv.InviteID == "" { + return + } + inv.Pin = nil + inv.ReceivedAt = s.now().UnixMilli() + s.remember(inv, stripPin(msg.Params, inv.ReceivedAt)) + s.publish(Event{Kind: EventInviteReceived, Invite: inv}) + + case "cluster:invite-canceled", "cluster:invite-expired": + var inv Invite + if err := decode(msg.Params, &inv); err != nil || inv.InviteID == "" { + return + } + inv.Pin = nil + if !s.forget(inv.InviteID) { + // Not one of ours — an outbound invite of this node's expiring. + return + } + s.publish(Event{Kind: EventInviteCleared, Invite: inv}) + } +} + +// remember adds an inbound invite to the pending set. A replacement invite +// carrying an id already held simply overwrites it, keeping its position. +func (s *Service) remember(inv Invite, raw json.RawMessage) { + s.mu.Lock() + defer s.mu.Unlock() + if _, exists := s.pending[inv.InviteID]; !exists { + s.arrived = append(s.arrived, inv.InviteID) + } + s.pending[inv.InviteID] = pendingInvite{invite: inv, raw: raw} +} + +// forget drops an invite from the pending set, reporting whether it was there. +func (s *Service) forget(inviteID string) bool { + s.mu.Lock() + defer s.mu.Unlock() + if _, ok := s.pending[inviteID]; !ok { + return false + } + delete(s.pending, inviteID) + for i, id := range s.arrived { + if id == inviteID { + s.arrived = append(s.arrived[:i], s.arrived[i+1:]...) + break + } + } + return true +} + +// solePending resolves "the invite that is pending" for a caller that named +// none, and reports precisely why it could not when there is not exactly one. +func (s *Service) solePending() (string, error) { + waiting := s.Pending() + switch len(waiting) { + case 0: + return "", ErrNoPendingInvite + case 1: + return waiting[0].InviteID, nil + default: + sorted := append([]Invite(nil), waiting...) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].ReceivedAt < sorted[j].ReceivedAt }) + return "", &AmbiguousInviteError{Invites: sorted} + } +} + +// call issues one broker request and decodes the Invite it answers with, +// keeping the manager's own JSON alongside the decode. +func (s *Service) call(ctx context.Context, method string, params any) (Result, error) { + msg, err := s.broker.Call(ctx, method, params) + if err != nil { + return Result{}, err + } + if msg == nil { + return Result{}, fmt.Errorf("%s returned no response", method) + } + var inv Invite + if err := decode(msg.Result, &inv); err != nil { + return Result{}, fmt.Errorf("decode %s: %w", method, err) + } + return Result{Raw: msg.Result, Invite: inv}, nil +} + +// unanswerable reports whether a respond-to-invite error means the invite can +// never be answered: -32001 unknown invite id, -32002 invalid invite state. +func unanswerable(err error) bool { + var rpcErr *rpc.RPCError + if !errors.As(err, &rpcErr) { + return false + } + return rpcErr.Code == -32001 || rpcErr.Code == -32002 +} + +func decode(raw json.RawMessage, v any) error { + if len(raw) == 0 { + return nil + } + return json.Unmarshal(raw, v) +} diff --git a/services/nvpair-tui/pairing/service_test.go b/services/nvpair-tui/pairing/service_test.go new file mode 100644 index 00000000..e1ebf97b --- /dev/null +++ b/services/nvpair-tui/pairing/service_test.go @@ -0,0 +1,579 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package pairing + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "log" + "log/slog" + "strings" + "sync" + "testing" + "time" + + "nvpair-tui/rpc" +) + +// testPIN is the secret every leak assertion in this file looks for. It is +// distinctive enough that a substring match cannot fire by accident. +const testPIN = "402199" + +// fakeBroker stands in for the TUI's broker client, recording what was asked +// and answering with whatever the test scripted. +type fakeBroker struct { + mu sync.Mutex + calls []brokerCall + answers map[string]func(params any) (json.RawMessage, error) +} + +type brokerCall struct { + method string + params map[string]any +} + +func newFakeBroker() *fakeBroker { + return &fakeBroker{answers: map[string]func(any) (json.RawMessage, error){}} +} + +func (f *fakeBroker) answer(method string, fn func(params any) (json.RawMessage, error)) { + f.mu.Lock() + defer f.mu.Unlock() + f.answers[method] = fn +} + +// reply is the common case: a fixed JSON result for a method. +func (f *fakeBroker) reply(method, result string) { + f.answer(method, func(any) (json.RawMessage, error) { + return json.RawMessage(result), nil + }) +} + +func (f *fakeBroker) Call(_ context.Context, method string, params any) (*rpc.Message, error) { + f.mu.Lock() + recorded := brokerCall{method: method} + if m, ok := params.(map[string]any); ok { + recorded.params = m + } + f.calls = append(f.calls, recorded) + fn := f.answers[method] + f.mu.Unlock() + if fn == nil { + return nil, fmt.Errorf("fake broker has no answer for %q", method) + } + result, err := fn(params) + if err != nil { + return nil, err + } + return &rpc.Message{JSONRPC: "2.0", Result: result}, nil +} + +func (f *fakeBroker) lastParams(t *testing.T, method string) map[string]any { + t.Helper() + f.mu.Lock() + defer f.mu.Unlock() + for i := len(f.calls) - 1; i >= 0; i-- { + if f.calls[i].method == method { + return f.calls[i].params + } + } + t.Fatalf("%s was never called; calls = %+v", method, f.calls) + return nil +} + +// inviteReceived builds the notification the cluster manager pushes to a node +// that has been invited. It never carries a PIN. +func inviteReceived(inviteID, fromName string) *rpc.Message { + return &rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-received", + Params: json.RawMessage(fmt.Sprintf( + `{"inviteId":%q,"fromNodeId":"NODE-A","fromNodeUuid":"uuid-a","fromNodeName":%q,`+ + `"clusterId":"cluster-xyz","pin":null,"state":"pending","createdAt":1716998400000,"respondedAt":null}`, + inviteID, fromName)), + } +} + +func TestInviteSplitsHostPortAndCarriesThePIN(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:invite-node", + `{"inviteId":"inv-1","state":"pending","pin":"`+testPIN+`","fromNodeName":"Lab desk A"}`) + svc := NewService(broker) + + res, err := svc.Invite(context.Background(), InviteRequest{Address: "10.0.0.5:14399"}) + if err != nil { + t.Fatalf("Invite: %v", err) + } + if res.Invite.PIN() != testPIN { + t.Errorf("PIN = %q, want the manager's own", res.Invite.PIN()) + } + // The result is relayed verbatim so a caller sees what the manager said, + // not a re-encoding of the fields this package happens to model. + if !bytes.Contains(res.Raw, []byte(`"inviteId":"inv-1"`)) { + t.Errorf("raw result = %s, want the manager's own JSON", res.Raw) + } + + params := broker.lastParams(t, "cluster:invite-node") + if params["address"] != "10.0.0.5" { + t.Errorf("address = %v, want the host split off", params["address"]) + } + if params["port"] != 14399 { + t.Errorf("port = %v, want the port split into its own field", params["port"]) + } +} + +func TestInviteParams(t *testing.T) { + tests := []struct { + name string + req InviteRequest + want map[string]any + wantErr bool + errFrags []string + }{ + { + name: "a bare host lets the manager append its own port", + req: InviteRequest{Address: "gpu-box.tail1234.ts.net"}, + want: map[string]any{"address": "gpu-box.tail1234.ts.net"}, + }, + { + name: "an explicit port wins over one embedded in the address", + req: InviteRequest{Address: "10.0.0.5:1111", Port: 2222}, + want: map[string]any{"address": "10.0.0.5:1111", "port": 2222}, + }, + { + name: "a node id is passed through as the target identity", + req: InviteRequest{Address: "10.0.0.5", NodeID: "uuid-b"}, + want: map[string]any{"address": "10.0.0.5", "nodeId": "uuid-b"}, + }, + { + name: "a bracketed IPv6 host:port splits like any other", + req: InviteRequest{Address: "[fd00::1]:14321"}, + want: map[string]any{"address": "fd00::1", "port": 14321}, + }, + { + name: "an empty address is rejected before any call", + req: InviteRequest{Address: " "}, + wantErr: true, + errFrags: []string{"address is required"}, + }, + { + name: "an out-of-range port is rejected", + req: InviteRequest{Address: "10.0.0.5", Port: 70000}, + wantErr: true, + errFrags: []string{"out of range"}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got, err := tc.req.params() + if tc.wantErr { + if err == nil { + t.Fatalf("params() = %v, want an error", got) + } + for _, frag := range tc.errFrags { + if !strings.Contains(err.Error(), frag) { + t.Errorf("error %q does not mention %q", err, frag) + } + } + return + } + if err != nil { + t.Fatalf("params(): %v", err) + } + if fmt.Sprint(got) != fmt.Sprint(tc.want) { + t.Errorf("params() = %v, want %v", got, tc.want) + } + }) + } +} + +func TestInviteReportsARejectionAsAResultNotAnError(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:invite-node", `{"inviteId":"inv-1","state":"rejected","reason":"already-clustered","pin":null}`) + svc := NewService(broker) + events, stop := svc.Subscribe() + defer stop() + + res, err := svc.Invite(context.Background(), InviteRequest{Address: "10.0.0.5"}) + if err != nil { + t.Fatalf("a rejection is a result, not an error: %v", err) + } + if res.Invite.State != StateRejected || res.Invite.Reason != "already-clustered" { + t.Fatalf("invite = %+v, want a rejection carrying its reason", res.Invite) + } + if ev := nextEvent(t, events); ev.Kind != EventInviteRejected { + t.Errorf("event kind = %q, want %q", ev.Kind, EventInviteRejected) + } +} + +func TestPendingTracksInboundInvitesAndStripsThePin(t *testing.T) { + svc := NewService(newFakeBroker()) + + if got := svc.Pending(); len(got) != 0 { + t.Fatalf("a fresh service has %d pending invites, want none", len(got)) + } + + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + svc.HandleNotification(inviteReceived("inv-2", "Lab desk B")) + + pending := svc.Pending() + if len(pending) != 2 { + t.Fatalf("pending = %d invites, want 2", len(pending)) + } + if pending[0].InviteID != "inv-1" || pending[1].InviteID != "inv-2" { + t.Errorf("pending order = %s,%s, want arrival order", pending[0].InviteID, pending[1].InviteID) + } + if pending[0].ReceivedAt == 0 { + t.Error("an inbound invite must be stamped with this node's own clock for its age") + } + for _, inv := range pending { + if inv.Pin != nil { + t.Errorf("invite %s carries a pin; an inbound invite never does", inv.InviteID) + } + } + + // The raw form keeps every field the manager sent, minus the pin. + raw := svc.PendingRaw() + if len(raw) != 2 { + t.Fatalf("PendingRaw = %d invites, want 2", len(raw)) + } + var fields map[string]any + if err := json.Unmarshal(raw[0], &fields); err != nil { + t.Fatalf("decode raw pending invite: %v", err) + } + if _, ok := fields["pin"]; ok { + t.Error("PendingRaw kept the pin member; it must be stripped") + } + if fields["clusterId"] != "cluster-xyz" { + t.Errorf("PendingRaw dropped clusterId (%v); the manager's own fields must survive", fields["clusterId"]) + } + if fields["receivedAt"] == nil { + t.Error("PendingRaw did not stamp receivedAt") + } +} + +func TestAWithdrawnInviteStopsBeingPending(t *testing.T) { + for _, method := range []string{"cluster:invite-canceled", "cluster:invite-expired"} { + t.Run(method, func(t *testing.T) { + svc := NewService(newFakeBroker()) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + events, stop := svc.Subscribe() + defer stop() + + svc.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: method, + Params: json.RawMessage(`{"inviteId":"inv-1","state":"expired","fromNodeName":"Lab desk A"}`), + }) + if got := svc.Pending(); len(got) != 0 { + t.Fatalf("pending = %d, want the withdrawn invite gone", len(got)) + } + if ev := nextEvent(t, events); ev.Kind != EventInviteCleared { + t.Errorf("event kind = %q, want %q", ev.Kind, EventInviteCleared) + } + }) + } +} + +func TestAWithdrawalForSomebodyElsesInvitePublishesNothing(t *testing.T) { + svc := NewService(newFakeBroker()) + events, stop := svc.Subscribe() + defer stop() + + // An outbound invite of this node's expiring: not in the pending set, so + // there is nothing to clear and nothing to say. + svc.HandleNotification(&rpc.Message{ + JSONRPC: "2.0", + Method: "cluster:invite-expired", + Params: json.RawMessage(`{"inviteId":"inv-outbound","state":"expired"}`), + }) + select { + case ev := <-events: + t.Fatalf("published %+v for an invite this node was not holding", ev) + case <-time.After(50 * time.Millisecond): + } +} + +func TestRespondResolvesTheInviteWhenExactlyOneIsPending(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-1","state":"paired","fromNodeName":"Lab desk A"}`) + svc := NewService(broker) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + res, err := svc.Respond(context.Background(), "", true, testPIN) + if err != nil { + t.Fatalf("Respond: %v", err) + } + if res.Invite.State != StatePaired { + t.Errorf("state = %q, want paired", res.Invite.State) + } + params := broker.lastParams(t, "cluster:respond-to-invite") + if params["inviteId"] != "inv-1" { + t.Errorf("inviteId = %v, want the one pending invite resolved", params["inviteId"]) + } + if params["pin"] != testPIN { + t.Errorf("pin = %v, want it forwarded to the manager", params["pin"]) + } + if got := svc.Pending(); len(got) != 0 { + t.Errorf("an answered invite is still pending: %+v", got) + } +} + +func TestRespondWithNoPendingInvite(t *testing.T) { + svc := NewService(newFakeBroker()) + _, err := svc.Respond(context.Background(), "", true, testPIN) + if !errors.Is(err, ErrNoPendingInvite) { + t.Fatalf("err = %v, want ErrNoPendingInvite", err) + } +} + +func TestRespondWithSeveralPendingNamesThemAll(t *testing.T) { + svc := NewService(newFakeBroker()) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + svc.HandleNotification(inviteReceived("inv-2", "Lab desk B")) + + _, err := svc.Respond(context.Background(), "", true, testPIN) + var ambiguous *AmbiguousInviteError + if !errors.As(err, &ambiguous) { + t.Fatalf("err = %v, want *AmbiguousInviteError", err) + } + if len(ambiguous.Invites) != 2 { + t.Fatalf("named %d invites, want 2", len(ambiguous.Invites)) + } + for _, want := range []string{"inv-1", "inv-2", "Lab desk A", "Lab desk B", "--invite"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("message %q does not mention %q", err, want) + } + } +} + +func TestRespondDropsAnInviteTheManagerCanNoLongerAnswer(t *testing.T) { + for _, code := range []int{-32001, -32002} { + t.Run(fmt.Sprintf("code %d", code), func(t *testing.T) { + broker := newFakeBroker() + broker.answer("cluster:respond-to-invite", func(any) (json.RawMessage, error) { + return nil, &rpc.RPCError{Code: code, Message: "gone"} + }) + svc := NewService(broker) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + if _, err := svc.Respond(context.Background(), "", true, testPIN); err == nil { + t.Fatal("Respond must surface the manager's error") + } + if got := svc.Pending(); len(got) != 0 { + t.Errorf("an unanswerable invite stayed pending: %+v", got) + } + }) + } +} + +func TestRespondKeepsAnInviteAfterATransientFailure(t *testing.T) { + broker := newFakeBroker() + broker.answer("cluster:respond-to-invite", func(any) (json.RawMessage, error) { + return nil, &rpc.RPCError{Code: -32603, Message: "internal error"} + }) + svc := NewService(broker) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + if _, err := svc.Respond(context.Background(), "", true, testPIN); err == nil { + t.Fatal("Respond must surface the manager's error") + } + if got := svc.Pending(); len(got) != 1 { + t.Errorf("pending = %d, want the invite kept so it can be retried", len(got)) + } +} + +func TestRespondDeclineSendsNoPin(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-1","state":"declined"}`) + svc := NewService(broker) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + if _, err := svc.Respond(context.Background(), "inv-1", false, testPIN); err != nil { + t.Fatalf("Respond: %v", err) + } + params := broker.lastParams(t, "cluster:respond-to-invite") + if _, ok := params["pin"]; ok { + t.Error("a decline carried a pin; there is nothing to prove on a decline") + } +} + +func TestRespondReportsAWrongPinAsAResult(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-1","state":"failed","reason":"incorrect-pin"}`) + svc := NewService(broker) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + res, err := svc.Respond(context.Background(), "", true, "000000") + if err != nil { + t.Fatalf("a wrong PIN is a result, not an error: %v", err) + } + if res.Invite.State != StateFailed || res.Invite.Reason != ReasonIncorrectPin { + t.Errorf("invite = %+v, want failed/incorrect-pin", res.Invite) + } +} + +func TestMembersJoinsIdentityAndRoster(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:get-node-id", + `{"nodeUuid":"uuid-a","nodeId":"NODE-A","name":"Lab desk A","clusterId":"cluster-xyz","clusterFriendlyName":"Lab 3 desks"}`) + broker.reply("nodes:get-initial", + `{"nodes":[{"id":"NODE-B","nodeUuid":"uuid-b","name":"Lab desk B","ipAddress":"10.0.0.5","port":14321,"state":"member"}]}`) + svc := NewService(broker) + + got, err := svc.Members(context.Background()) + if err != nil { + t.Fatalf("Members: %v", err) + } + if got.ClusterID != "cluster-xyz" || got.ClusterFriendlyName != "Lab 3 desks" { + t.Errorf("identity = %+v, want the cluster it reported", got) + } + if len(got.Members) != 1 || got.Members[0].ID != "NODE-B" || got.Members[0].IPAddress != "10.0.0.5" { + t.Errorf("members = %+v, want the roster", got.Members) + } +} + +func TestMembersOfAnUnclusteredNodeIsAnEmptyList(t *testing.T) { + broker := newFakeBroker() + broker.reply("cluster:get-node-id", `{"nodeUuid":"uuid-a","nodeId":"NODE-A","name":"Lab desk A","clusterId":""}`) + broker.reply("nodes:get-initial", `{"nodes":null}`) + svc := NewService(broker) + + got, err := svc.Members(context.Background()) + if err != nil { + t.Fatalf("Members: %v", err) + } + if got.ClusterID != "" { + t.Errorf("clusterId = %q, want empty", got.ClusterID) + } + if got.Members == nil || len(got.Members) != 0 { + t.Errorf("members = %#v, want an empty list rather than null", got.Members) + } +} + +func TestAwaitPendingReturnsAnInviteThatIsAlreadyWaiting(t *testing.T) { + svc := NewService(newFakeBroker()) + svc.HandleNotification(inviteReceived("inv-1", "Lab desk A")) + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + inv, err := svc.AwaitPending(ctx) + if err != nil { + t.Fatalf("AwaitPending: %v", err) + } + if inv.InviteID != "inv-1" { + t.Errorf("inviteId = %q, want inv-1", inv.InviteID) + } +} + +func TestAwaitPendingWakesOnAnInviteThatArrivesLater(t *testing.T) { + svc := NewService(newFakeBroker()) + go func() { + time.Sleep(30 * time.Millisecond) + svc.HandleNotification(inviteReceived("inv-late", "Lab desk A")) + }() + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + inv, err := svc.AwaitPending(ctx) + if err != nil { + t.Fatalf("AwaitPending: %v", err) + } + if inv.InviteID != "inv-late" { + t.Errorf("inviteId = %q, want inv-late", inv.InviteID) + } +} + +func TestAwaitPendingHonoursItsDeadline(t *testing.T) { + svc := NewService(newFakeBroker()) + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + if _, err := svc.AwaitPending(ctx); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("err = %v, want the deadline to expire", err) + } +} + +// TestASlowSubscriberCannotStallAPairing fills a subscriber's buffer and +// checks that the pairing it is not reading about still completes. +func TestASlowSubscriberCannotStallAPairing(t *testing.T) { + svc := NewService(newFakeBroker()) + _, stop := svc.Subscribe() // subscribed, never read + defer stop() + + done := make(chan struct{}) + go func() { + defer close(done) + for i := 0; i < eventBuffer*3; i++ { + svc.HandleNotification(inviteReceived(fmt.Sprintf("inv-%d", i), "Lab desk A")) + } + }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("a subscriber that stopped reading blocked the pairing service") + } +} + +// TestThePINNeverReachesTheLog drives a whole pairing — both sides of it — +// through a service whose logger writes into a buffer, and asserts the PIN is +// nowhere in what was logged. +// +// The default logger is replaced rather than applog.SetOutput, which is a +// once-only hook that a second test cannot re-arm; a marker line proves the +// buffer really is capturing, so the assertion cannot pass vacuously. +func TestThePINNeverReachesTheLog(t *testing.T) { + var logged bytes.Buffer + previous := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&logged, &slog.HandlerOptions{Level: slog.LevelDebug}))) + previousWriter := log.Writer() + log.SetOutput(&logged) + t.Cleanup(func() { + slog.SetDefault(previous) + log.SetOutput(previousWriter) + }) + slog.Info("capture-marker") + + broker := newFakeBroker() + broker.reply("cluster:invite-node", `{"inviteId":"inv-1","state":"pending","pin":"`+testPIN+`"}`) + broker.reply("cluster:invite-status", `{"inviteId":"inv-1","state":"pending","pin":"`+testPIN+`"}`) + broker.reply("cluster:respond-to-invite", `{"inviteId":"inv-2","state":"paired"}`) + svc := NewService(broker) + + if _, err := svc.Invite(context.Background(), InviteRequest{Address: "10.0.0.5"}); err != nil { + t.Fatalf("Invite: %v", err) + } + if _, err := svc.InviteStatus(context.Background(), "inv-1"); err != nil { + t.Fatalf("InviteStatus: %v", err) + } + svc.HandleNotification(inviteReceived("inv-2", "Lab desk A")) + if _, err := svc.Respond(context.Background(), "", true, testPIN); err != nil { + t.Fatalf("Respond: %v", err) + } + + captured := logged.String() + if !strings.Contains(captured, "capture-marker") { + t.Fatal("the log buffer captured nothing, so this assertion proves nothing") + } + if strings.Contains(captured, testPIN) { + t.Errorf("the pairing PIN reached the log:\n%s", captured) + } +} + +// nextEvent takes the next pairing event, failing rather than hanging. +func nextEvent(t *testing.T, events <-chan Event) Event { + t.Helper() + select { + case ev, ok := <-events: + if !ok { + t.Fatal("the event channel closed") + } + return ev + case <-time.After(2 * time.Second): + t.Fatal("no pairing event arrived") + return Event{} + } +} diff --git a/services/nvpair-tui/ui/cluster.go b/services/nvpair-tui/ui/cluster.go index ac334edf..fa587015 100644 --- a/services/nvpair-tui/ui/cluster.go +++ b/services/nvpair-tui/ui/cluster.go @@ -5,10 +5,10 @@ package ui import ( "fmt" - "net" "strconv" "strings" + "nvpair-tui/pairing" "nvpair-tui/rpc" "github.com/charmbracelet/bubbles/key" @@ -36,15 +36,6 @@ type clusterNode struct { State string `json:"state"` } -// clusterInvite is the broker-facing view of a pairing session -// (cluster:invite-received push / cluster:invite-node result). -type clusterInvite struct { - InviteID string `json:"inviteId"` - FromNodeName string `json:"fromNodeName"` - Pin *string `json:"pin"` - State string `json:"state"` -} - type clusterInputMode int const ( @@ -57,11 +48,15 @@ const ( // roster (live from nodes:changed), outbound invites (showing the PIN to // read to the joiner), and inbound invites (entering the PIN to accept). type clusterView struct { - client *rpc.Client + client *rpc.Client + // pairs is the process-wide pairing service. It owns which invites are + // waiting and it is what the control socket drives too, which is how an + // invite created or answered from a script shows up on this tab. + pairs *pairing.Service + events <-chan pairing.Event table table.Model identity clusterIdentity nodes []clusterNode - pending *clusterInvite // most recent inbound invite awaiting a response input textinput.Model mode clusterInputMode status string @@ -79,12 +74,13 @@ type clusterNodesMsg struct { err error } +// clusterActionMsg carries the outcome of a non-pairing cluster action fired +// from this tab (remove a member, leave the cluster). A pairing outcome +// arrives as a pairing event instead, so that one driven from the control +// socket is reported here identically to one driven from the keyboard. type clusterActionMsg struct { - what string - pin string - rejected bool - reason string - err error + what string + err error } var ( @@ -95,9 +91,10 @@ var ( clLeaveKey = key.NewBinding(key.WithKeys("L"), key.WithHelp("L", "leave cluster")) ) -func newClusterView(client *rpc.Client) *clusterView { +func newClusterView(client *rpc.Client, pairs *pairing.Service) *clusterView { ti := textinput.New() - v := &clusterView{client: client, input: ti} + events, _ := pairs.Subscribe() + v := &clusterView{client: client, pairs: pairs, events: events, input: ti} v.table = newTable(nil) return v } @@ -105,7 +102,7 @@ func newClusterView(client *rpc.Client) *clusterView { func (v *clusterView) Title() string { return "Cluster" } func (v *clusterView) Init() tea.Cmd { - return tea.Batch(v.identityCmd(), v.nodesCmd()) + return tea.Batch(v.identityCmd(), v.nodesCmd(), waitForPairingEvent(v.events)) } func (v *clusterView) identityCmd() tea.Cmd { @@ -164,14 +161,13 @@ func (v *clusterView) Update(msg tea.Msg) tea.Cmd { case clusterActionMsg: if msg.err != nil { v.status = msg.what + " failed: " + msg.err.Error() - } else if msg.rejected { - v.status = fmt.Sprintf("invite rejected (%s) - remove the existing relationship first", rejectReason(msg.reason)) - } else if msg.pin != "" { - v.status = fmt.Sprintf("invite sent - PIN %s (read it to the joining node)", msg.pin) } else { v.status = msg.what + " ok" } return nil + case pairingEventMsg: + v.applyPairingEvent(msg.Event) + return waitForPairingEvent(v.events) case NotificationMsg: return v.handleNotification(msg.Msg) case tea.KeyMsg: @@ -194,12 +190,10 @@ func (v *clusterView) handleNotification(msg *rpc.Message) tea.Cmd { } _ = decodeParams(msg.Params, &r) v.identity.ClusterID = r.ClusterID - case "cluster:invite-received": - var inv clusterInvite - _ = decodeParams(msg.Params, &inv) - v.pending = &inv - v.status = "invite received from " + inv.FromNodeName + " - press a to accept, d to decline" } + // The pairing pushes (cluster:invite-received and the invite-canceled / + // -expired that retract one) are folded into the pairing service by the + // process's notification fan-out; this tab hears about them as events. return nil } @@ -222,7 +216,7 @@ func (v *clusterView) handleKey(msg tea.KeyMsg) tea.Cmd { v.beginInput(clusterInputAddress, "host (or host:port; default 14321)") return textinput.Blink case key.Matches(msg, clAcceptKey): - if v.pending != nil { + if len(v.pairs.Pending()) > 0 { v.beginInput(clusterInputPin, "PIN from inviting node") return textinput.Blink } @@ -261,30 +255,14 @@ func (v *clusterView) submit() tea.Cmd { v.status = "address required" return nil } - // nvpair-cluster-manager treats "address" as a bare host and appends the - // port itself (default 14321). If the operator typed host:port, split - // it so the port lands in the manager's separate int field instead of - // being glued onto the host (which would dial [host:port]:14321). - params := map[string]any{"address": val} - if host, portStr, err := net.SplitHostPort(val); err == nil { - if port, perr := strconv.Atoi(portStr); perr == nil { - params["address"] = host - params["port"] = port - } - } + // The pairing service splits a typed "host:port" into the manager's + // separate address and port fields; a bare host gets the manager's own + // default port appended. v.status = "inviting " + val + "..." - return inviteNodeCmd(v.client, params, func(res inviteNodeResult, err error) tea.Msg { - if err != nil { - return clusterActionMsg{what: "invite", err: err} - } - if res.State == "rejected" { - return clusterActionMsg{what: "invite", rejected: true, reason: res.Reason} - } - pin := "" - if res.Pin != nil { - pin = *res.Pin - } - return clusterActionMsg{what: "invite", pin: pin} + return inviteCmd(v.pairs, pairing.InviteRequest{Address: val}, func(pairing.Invite, error) tea.Msg { + // The status line is written from the pairing event, so that an + // invite from the control socket reads the same as this one. + return nil }) case clusterInputPin: return v.respondToInvite(true, val) @@ -292,22 +270,79 @@ func (v *clusterView) submit() tea.Cmd { return nil } +// respondToInvite answers the invite this tab is showing. Which one that is +// comes from the pairing service, so the keyboard and the control socket agree +// on it even when several arrived. func (v *clusterView) respondToInvite(accept bool, pin string) tea.Cmd { - if v.pending == nil { + waiting := v.pairs.Pending() + if len(waiting) == 0 { + v.status = "no invite is waiting for an answer" return nil } - params := map[string]any{"inviteId": v.pending.InviteID, "accept": accept} - if accept && pin != "" { - params["pin"] = pin + return respondCmd(v.pairs, waiting[0].InviteID, accept, pin) +} + +// applyPairingEvent renders one pairing state change on the status line, +// whichever driver caused it. An invite created over the control socket shows +// its PIN here exactly like one created with `i`, and an accept made there +// clears a PIN prompt this tab left open. +func (v *clusterView) applyPairingEvent(ev pairing.Event) { + switch ev.Kind { + case pairing.EventInviteSent: + v.status = fmt.Sprintf("invite sent - PIN %s (read it to the joining node)", ev.Invite.PIN()) + case pairing.EventInviteRejected: + v.status = fmt.Sprintf("invite rejected (%s) - remove the existing relationship first", rejectReason(ev.Invite.Reason)) + case pairing.EventInviteFailed: + v.status = "invite failed: " + ev.Err.Error() + case pairing.EventInviteReceived: + v.status = "invite received from " + ev.Invite.FromNodeName + " - press a to accept, d to decline" + case pairing.EventInviteCleared: + v.status = fmt.Sprintf("invite from %s %s", ev.Invite.FromNodeName, ev.Invite.State) + v.closePinPromptIfIdle() + case pairing.EventResponded: + v.status = respondedStatus(ev) + v.closePinPromptIfIdle() + case pairing.EventRespondFailed: + what := "decline invite" + if ev.Accept { + what = "accept invite" + } + v.status = what + " failed: " + ev.Err.Error() + v.closePinPromptIfIdle() + } +} + +// closePinPromptIfIdle dismisses a PIN prompt once nothing is waiting for an +// answer any more — the case where a script accepted or declined the very +// invite the operator was still typing a PIN for. +func (v *clusterView) closePinPromptIfIdle() { + if v.mode == clusterInputPin && len(v.pairs.Pending()) == 0 { + v.cancelInput() + } +} + +// respondedStatus words the outcome of an answered invite. A wrong PIN is a +// successful response in a failed state rather than an error, so it is +// reported here and not on the failure path. +func respondedStatus(ev pairing.Event) string { + if !ev.Accept { + return "invite declined" } - v.pending = nil - what := "decline invite" - if accept { - what = "accept invite" + switch ev.Invite.State { + case pairing.StatePaired: + name := ev.Invite.FromNodeName + if name == "" { + name = "the inviting node" + } + return "paired with " + name + case pairing.StateFailed: + if ev.Invite.Reason == pairing.ReasonIncorrectPin { + return "incorrect PIN - ask for it again and retry" + } + return "pairing failed" + default: + return "accept invite: " + ev.Invite.State } - return call(v.client, "cluster:respond-to-invite", params, func(_ *rpc.Message, err error) tea.Msg { - return clusterActionMsg{what: what, err: err} - }) } // leaveCluster unjoins this node from its cluster (cluster:leave). The diff --git a/services/nvpair-tui/ui/invite.go b/services/nvpair-tui/ui/invite.go index 039c3430..8a8bff42 100644 --- a/services/nvpair-tui/ui/invite.go +++ b/services/nvpair-tui/ui/invite.go @@ -4,34 +4,65 @@ package ui import ( - "nvpair-tui/rpc" + "context" + + "nvpair-tui/pairing" tea "github.com/charmbracelet/bubbletea" ) -// inviteNodeResult is the decoded cluster:invite-node result the UI acts on: a -// PIN to display on success, or an explicit rejection (e.g. the target is -// already clustered) carrying its reason. No PIN accompanies a rejection. -type inviteNodeResult struct { - State string `json:"state"` - Pin *string `json:"pin"` - Reason string `json:"reason"` -} - -// inviteNodeCmd issues a single cluster:invite-node request and maps the -// decoded result (or error) into the caller's view message (shared by the -// Cluster and Nodes tabs so the decode lives in one place). +// inviteCmd starts one outbound pairing through the shared pairing service — +// the same call the control socket's pair:invite makes, so the Cluster tab and +// a script cannot diverge. It maps the outcome into the caller's view message +// (the Cluster and Nodes tabs each word their status line differently). // // There is no separate "create cluster" step: the backend auto-founds a // cluster of one when this node isn't clustered yet, so the invite is // the one authoritative call and the UI carries no membership orchestration. -func inviteNodeCmd(client *rpc.Client, params map[string]any, finish func(res inviteNodeResult, err error) tea.Msg) tea.Cmd { - return call(client, "cluster:invite-node", params, func(msg *rpc.Message, err error) tea.Msg { +// +// The Cluster tab's status line is driven by the service's event stream rather +// than by this result, so an invite started from the control socket lands +// there exactly like one started with `i`. +func inviteCmd(svc *pairing.Service, req pairing.InviteRequest, finish func(inv pairing.Invite, err error) tea.Msg) tea.Cmd { + return func() tea.Msg { + ctx, cancel := context.WithTimeout(context.Background(), callTimeout) + defer cancel() + res, err := svc.Invite(ctx, req) if err != nil { - return finish(inviteNodeResult{}, err) + return finish(pairing.Invite{}, err) } - var r inviteNodeResult - _ = decodeParams(msg.Result, &r) - return finish(r, nil) - }) + return finish(res.Invite, nil) + } } + +// respondCmd answers an inbound invite through the shared pairing service. +// inviteID may be empty, in which case the service resolves the single pending +// invite or reports why it could not. +// +// Nothing is returned into the update loop: the outcome, success or failure, +// reaches the view as a pairing event, which is also how an accept made over +// the control socket clears this tab's prompt. +func respondCmd(svc *pairing.Service, inviteID string, accept bool, pin string) tea.Cmd { + return func() tea.Msg { + ctx, cancel := context.WithTimeout(context.Background(), callTimeout) + defer cancel() + _, _ = svc.Respond(ctx, inviteID, accept, pin) + return nil + } +} + +// waitForPairingEvent blocks on the next pairing event and re-arms itself, the +// same shape as waitForNotification. A closed channel ends the pump. +func waitForPairingEvent(events <-chan pairing.Event) tea.Cmd { + return func() tea.Msg { + ev, ok := <-events + if !ok { + return nil + } + return pairingEventMsg{Event: ev} + } +} + +// pairingEventMsg carries one pairing state change into the update loop, +// whether this TUI's keyboard or the control socket caused it. +type pairingEventMsg struct{ Event pairing.Event } diff --git a/services/nvpair-tui/ui/model.go b/services/nvpair-tui/ui/model.go index 971ab063..805c7aaf 100644 --- a/services/nvpair-tui/ui/model.go +++ b/services/nvpair-tui/ui/model.go @@ -23,10 +23,11 @@ const chromeHeight = 3 // header showing broker status, and a footer of contextual help. It owns // the broker notification loop and routes messages to the views. type Model struct { - client *rpc.Client - logCh <-chan string - keys globalKeyMap - help help.Model + client *rpc.Client + notifications <-chan *rpc.Message + logCh <-chan string + keys globalKeyMap + help help.Model views []View active int @@ -39,22 +40,23 @@ type Model struct { showFullHelp bool } -// New builds the root model over a connected broker client, the broker's +// New builds the root model over the program's dependencies, the broker's // captured stderr line channel, and the set of views (tabs) to present, // in tab order. -func New(client *rpc.Client, logCh <-chan string, views []View) Model { +func New(deps Deps, logCh <-chan string, views []View) Model { return Model{ - client: client, - logCh: logCh, - keys: newGlobalKeyMap(), - help: help.New(), - views: views, + client: deps.Client, + notifications: deps.Notifications, + logCh: logCh, + keys: newGlobalKeyMap(), + help: help.New(), + views: views, } } // Init starts each view and arms the broker notification + log loops. func (m Model) Init() tea.Cmd { - cmds := []tea.Cmd{waitForNotification(m.client), waitForLog(m.logCh)} + cmds := []tea.Cmd{waitForNotification(m.notifications), waitForLog(m.logCh)} for _, v := range m.views { if c := v.Init(); c != nil { cmds = append(cmds, c) @@ -106,7 +108,7 @@ func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { m.brokerVersion = readyVersion(msg.Msg) } cmds := m.broadcast(msg) - cmds = append(cmds, waitForNotification(m.client)) + cmds = append(cmds, waitForNotification(m.notifications)) return m, tea.Batch(cmds...) case DisconnectedMsg: diff --git a/services/nvpair-tui/ui/nodes.go b/services/nvpair-tui/ui/nodes.go index 8c8db2e7..c206f257 100644 --- a/services/nvpair-tui/ui/nodes.go +++ b/services/nvpair-tui/ui/nodes.go @@ -8,6 +8,7 @@ import ( "strconv" "time" + "nvpair-tui/pairing" "nvpair-tui/rpc" "github.com/charmbracelet/bubbles/key" @@ -44,7 +45,10 @@ func (n availableNode) key() string { // subscribe and then pushes discovery:nodes-changed (a full list) on // every change. type nodesView struct { - client *rpc.Client + client *rpc.Client + // pairs is the shared pairing service, so an invite sent from this tab is + // the same call the Cluster tab and the control socket make. + pairs *pairing.Service table table.Model nodes []availableNode status string @@ -66,8 +70,8 @@ type nodeInviteMsg struct { var niInviteKey = key.NewBinding(key.WithKeys("i"), key.WithHelp("i", "invite to cluster")) -func newNodesView(client *rpc.Client) *nodesView { - v := &nodesView{client: client} +func newNodesView(client *rpc.Client, pairs *pairing.Service) *nodesView { + v := &nodesView{client: client, pairs: pairs} v.table = newTable(nil) return v } @@ -160,21 +164,17 @@ func (v *nodesView) inviteSelected() tea.Cmd { } // Identify the invite target by its stable UUID (the address is still the // dial target); the manager stamps this as the invite's target identity. - params := map[string]any{"address": n.IPAddress, "nodeId": n.key()} + req := pairing.InviteRequest{Address: n.IPAddress, NodeID: n.key()} name := n.Name v.status = "inviting " + name + "..." - return inviteNodeCmd(v.client, params, func(res inviteNodeResult, err error) tea.Msg { + return inviteCmd(v.pairs, req, func(inv pairing.Invite, err error) tea.Msg { if err != nil { return nodeInviteMsg{name: name, err: err} } - if res.State == "rejected" { - return nodeInviteMsg{name: name, rejected: true, reason: res.Reason} + if inv.State == pairing.StateRejected { + return nodeInviteMsg{name: name, rejected: true, reason: inv.Reason} } - pin := "" - if res.Pin != nil { - pin = *res.Pin - } - return nodeInviteMsg{name: name, pin: pin} + return nodeInviteMsg{name: name, pin: inv.PIN()} }) } diff --git a/services/nvpair-tui/ui/rpccmd.go b/services/nvpair-tui/ui/rpccmd.go index 007f56dd..df6a894d 100644 --- a/services/nvpair-tui/ui/rpccmd.go +++ b/services/nvpair-tui/ui/rpccmd.go @@ -47,9 +47,9 @@ func waitForLog(lines <-chan string) tea.Cmd { // waitForNotification blocks on the next broker push and delivers it as a // NotificationMsg, re-arming itself after each one (the model returns this // command again from Update). A closed channel yields DisconnectedMsg. -func waitForNotification(client *rpc.Client) tea.Cmd { +func waitForNotification(pushes <-chan *rpc.Message) tea.Cmd { return func() tea.Msg { - msg, ok := <-client.Notifications() + msg, ok := <-pushes if !ok { return DisconnectedMsg{} } diff --git a/services/nvpair-tui/ui/ui.go b/services/nvpair-tui/ui/ui.go index 65015a7b..c5c8191c 100644 --- a/services/nvpair-tui/ui/ui.go +++ b/services/nvpair-tui/ui/ui.go @@ -7,20 +7,38 @@ import ( "bufio" "io" + "nvpair-tui/pairing" "nvpair-tui/rpc" tea "github.com/charmbracelet/bubbletea" ) -// Run builds the tabbed program over a connected broker client and the -// broker's stderr stream, and blocks until the user quits. The caller is -// responsible for shutting the broker down afterwards. -func Run(client *rpc.Client, stderr io.Reader) error { +// Deps is everything the tabbed program is built over. The broker's +// notification stream arrives as a channel rather than straight off the +// client, because the process fans those pushes out to the pairing service +// first — the Cluster tab is one consumer of pairing state, not its owner. +type Deps struct { + // Client issues broker requests. + Client *rpc.Client + // Notifications carries the broker's pushes. Closed when the broker goes + // away, which is how the UI learns it disconnected. + Notifications <-chan *rpc.Message + // Stderr is the broker's captured stderr, shown in the Logs tab. + Stderr io.Reader + // Pairing is the process-wide pairing service, shared with the control + // socket so an invite created from a script shows up here too. + Pairing *pairing.Service +} + +// Run builds the tabbed program over its dependencies and blocks until the +// user quits. The caller is responsible for shutting the broker down +// afterwards. +func Run(deps Deps) error { logCh := make(chan string, 2000) - go scanLines(stderr, logCh) + go scanLines(deps.Stderr, logCh) p := tea.NewProgram( - New(client, logCh, defaultViews(client)), + New(deps, logCh, defaultViews(deps)), tea.WithAltScreen(), ) _, err := p.Run() @@ -40,15 +58,16 @@ func scanLines(r io.Reader, out chan<- string) { } // defaultViews lists the tabs in display order. -func defaultViews(client *rpc.Client) []View { +func defaultViews(deps Deps) []View { + client := deps.Client return []View{ newHealthView(client), newErrorsView(client), - newNodesView(client), + newNodesView(client, deps.Pairing), newProxiesView(client), newWorkloadsView(client), newEnginesView(client), - newClusterView(client), + newClusterView(client, deps.Pairing), newManualView(client), newSettingsView(client), newLogsView(client), diff --git a/services/readme.md b/services/readme.md index ee3523d5..8fcd99b9 100644 --- a/services/readme.md +++ b/services/readme.md @@ -51,7 +51,7 @@ This tree builds thirteen Go binaries. `nvpair-ui-broker` is the parent service | `nvpair-node-settings` | Typed key-value store for per-node preferences. | | `nvpair-cluster-manager` | Node identity, PIN pairing, and the trusted-node store. | | `nvpair-job-scheduler` | Responsive scheduler combining total node queue depth across engines with smoothed GPU pressure. | -| `nvpair-tui` | Terminal interface for headless and SSH operation; launches and supervises its own broker. | +| `nvpair-tui` | Terminal interface for headless and SSH operation; launches and supervises its own broker, and serves a per-user control socket so pairing can also be driven by `nvpair-tui invite` / `accept` from a script. | Shared code lives in the local `shared/` Go module (imported as `nvpair-shared/…`, replaced via `replace nvpair-shared => ../shared`). It provides logging, wire types, JSON-RPC and IPC, discovery records, mDNS, network monitoring, stable node identity, application data paths, and cluster trust helpers. diff --git a/services/tests/fixtures/stubbroker/main.go b/services/tests/fixtures/stubbroker/main.go new file mode 100644 index 00000000..42c2b15c --- /dev/null +++ b/services/tests/fixtures/stubbroker/main.go @@ -0,0 +1,171 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Command stubbroker is a test fixture: the smallest nvpair-ui-broker a real +// nvpair-tui will accept as its parent. +// +// It exists because the real broker's ports are compiled-in constants — one +// broker per machine — while a pairing needs *two* independent nodes. So this +// spawns nothing but a real nvpair-cluster-manager on a port the test chose, +// and relays the one namespace pairing needs (cluster:* / nodes:*) between the +// TUI above and the manager below, in both directions and byte for byte. +// +// Everything the assertions turn on stays real: a real nvpair-tui process, its +// real control socket and subcommands, and a real PIN-authenticated EAP-NOOB +// exchange between two real cluster managers. +// +// It deliberately does not synthesize a response for a relayed request the +// cluster manager never answers — if the manager dies mid-pairing, that request +// simply has no reply and the caller waits out its own 35 s timeout. The real +// broker tracks its workers and fails such a call fast; reproducing that here +// would be more fixture than the test needs. So a subcommand in this test that +// takes 35 s has lost its cluster manager, and the manager's stderr (passed +// through to the test's) says why. +package main + +import ( + "bufio" + "encoding/json" + "flag" + "io" + "log" + "os" + "os/exec" + "strconv" + "strings" + "sync" + + "nvpair-shared/jsonrpc" +) + +// maxFrame matches the broker's own read buffer, so a large roster snapshot is +// never truncated mid-frame on its way through. +const maxFrame = 1024 * 1024 + +func main() { + managerPath := flag.String("cluster-manager-path", "", "path to the nvpair-cluster-manager binary") + configDir := flag.String("config-dir", "", "cluster identity and trusted-store directory") + port := flag.Int("port", 0, "inter-node pairing port for the cluster manager") + flag.Parse() + + if *managerPath == "" || *configDir == "" || *port == 0 { + log.Fatal("stubbroker needs --cluster-manager-path, --config-dir and --port") + } + + manager := exec.Command(*managerPath, "--config-dir", *configDir, "--port", strconv.Itoa(*port)) + manager.Stderr = os.Stderr + managerIn, err := manager.StdinPipe() + if err != nil { + log.Fatalf("cluster-manager stdin: %v", err) + } + managerOut, err := manager.StdoutPipe() + if err != nil { + log.Fatalf("cluster-manager stdout: %v", err) + } + if err := manager.Start(); err != nil { + log.Fatalf("start cluster-manager: %v", err) + } + + up := newLink(os.Stdin, os.Stdout) + down := newLink(managerOut, managerIn) + + // The TUI's header and the control socket's ping both wait on this. + up.write(&jsonrpc.Message{ + JSONRPC: "2.0", + Method: "app:ready", + Params: json.RawMessage(`{"version":"stub"}`), + }) + + // Manager to TUI. Results keep their ids and notifications pass through + // untouched — cluster:invite-received above all, which is the only way the + // invited side learns it has something to answer. + var pump sync.WaitGroup + pump.Add(1) + go func() { + defer pump.Done() + for { + msg, ok := down.read() + if !ok { + return + } + up.write(msg) + } + }() + + shutdown := func() { + _ = managerIn.Close() + _ = manager.Wait() + pump.Wait() + } + + for { + msg, ok := up.read() + if !ok { + // stdin EOF: the TUI is gone, so take the manager with us. + shutdown() + return + } + if !msg.IsRequest() { + continue + } + switch { + case msg.Method == "shutdown": + up.write(&jsonrpc.Message{JSONRPC: "2.0", ID: msg.ID, Result: json.RawMessage(`{"ok":true}`)}) + shutdown() + return + case msg.Method == "ping": + up.write(&jsonrpc.Message{JSONRPC: "2.0", ID: msg.ID, Result: json.RawMessage(`{"version":"stub","uptimeMs":0}`)}) + case strings.HasPrefix(msg.Method, "cluster:"), strings.HasPrefix(msg.Method, "nodes:"): + down.write(msg) + default: + // Every other broker method is out of this fixture's scope. The + // TUI renders an unknown-method error as a tab with no data, + // which is exactly right here. + up.write(&jsonrpc.Message{ + JSONRPC: "2.0", + ID: msg.ID, + Error: &jsonrpc.RPCError{Code: -32601, Message: "stubbroker relays cluster:* and nodes:* only"}, + }) + } + } +} + +// link is one newline-delimited JSON-RPC direction. Frames are relayed as +// decoded-and-re-encoded Messages rather than through the shared codec's typed +// helpers, because a relay has to be able to pass a *request* along, which +// those helpers deliberately do not expose. +type link struct { + scanner *bufio.Scanner + out io.Writer + mu sync.Mutex +} + +func newLink(r io.Reader, w io.Writer) *link { + scanner := bufio.NewScanner(r) + scanner.Buffer(make([]byte, 0, 64*1024), maxFrame) + return &link{scanner: scanner, out: w} +} + +// read returns the next frame, or false once the stream ends. A line that will +// not decode is skipped rather than treated as the end. +func (l *link) read() (*jsonrpc.Message, bool) { + for l.scanner.Scan() { + var msg jsonrpc.Message + if err := json.Unmarshal(l.scanner.Bytes(), &msg); err != nil { + continue + } + return &msg, true + } + return nil, false +} + +func (l *link) write(msg *jsonrpc.Message) { + data, err := json.Marshal(msg) + if err != nil { + return + } + data = append(data, '\n') + l.mu.Lock() + defer l.mu.Unlock() + _, _ = l.out.Write(data) +} diff --git a/services/tests/headless_pairing_test.go b/services/tests/headless_pairing_test.go new file mode 100644 index 00000000..9d1ae757 --- /dev/null +++ b/services/tests/headless_pairing_test.go @@ -0,0 +1,434 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Cross-process gate for pairing two machines with no keyboard: the flow an +// operator runs over SSH on a headless GPU box. +// +// Two real nvpair-tui processes come up, each serving its own control socket, +// each owning a real nvpair-cluster-manager. `nvpair-tui invite` on one prints +// a PIN, `nvpair-tui accept --pin` on the other returns paired, and `members` +// on both then lists the pair. Nothing about the pairing is simulated: it is +// the real PIN-authenticated EAP-NOOB exchange between two real managers. +// +// Two things are substituted, both for the same reason — every broker-owned +// port is a compiled-in constant, so only one broker can exist per machine and +// this test needs two nodes: +// +// - each TUI's broker is the stubbroker fixture, which spawns a real cluster +// manager on an ephemeral port and relays cluster:* / nodes:* verbatim; +// - each TUI runs under `script`, because it is a full-screen program that +// will not start without a terminal. +// +// The TUI process, its control socket, its subcommands, the cluster managers +// and the pairing are all real. +// +// Unix only: it drives nvpair-tui under script(1) and reaps it by process +// group, neither of which Windows has. SysProcAttr.Setpgid does not even +// compile there, hence the build tag rather than a runtime skip. + +//go:build !windows + +package tests + +import ( + "encoding/json" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "syscall" + "testing" + "time" +) + +// tuiNode is one headless machine: a real nvpair-tui under a pty, its control +// socket, and the cluster manager its stub broker owns. +type tuiNode struct { + t *testing.T + name string + socket string + port int + cmd *exec.Cmd + stdin io.WriteCloser + log *os.File +} + +// shortDir gives a directory whose path leaves room for a socket name. A macOS +// t.TempDir() sits under /var/folders/... and, with a socket name appended, +// passes the 104-byte sun_path limit — a real constraint of this endpoint, +// which is why the control socket cannot simply live in t.TempDir(). +func shortDir(t *testing.T, label string) string { + t.Helper() + dir, err := os.MkdirTemp("/tmp", "nvpair-"+label) + if err != nil { + t.Fatalf("temp dir: %v", err) + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) + return dir +} + +// ptyCommand wraps argv so it runs with a controlling terminal. nvpair-tui is +// a Bubble Tea program: with no tty it fails at startup with "could not open a +// new TTY", so there is no way to exercise the real binary without one. +func ptyCommand(t *testing.T, argv []string) *exec.Cmd { + t.Helper() + if _, err := exec.LookPath("script"); err != nil { + t.Skipf("script(1) is not on PATH, so no pty can be allocated for nvpair-tui: %v", err) + } + switch runtime.GOOS { + case "darwin": + // BSD script: script [-q] file command [args...] + return exec.Command("script", append([]string{"-q", "/dev/null"}, argv...)...) + default: + // util-linux script (Linux and the other unixes that ship it) takes + // the command as one string. + return exec.Command("script", "-qec", strings.Join(argv, " "), "/dev/null") + } +} + +// startTUINode brings up one headless machine and returns once its control +// socket answers. +func startTUINode(t *testing.T, name string) *tuiNode { + t.Helper() + base := shortDir(t, name) + node := &tuiNode{ + t: t, + name: name, + socket: filepath.Join(base, "tui.sock"), + port: freePort(t), + } + + brokerArgs := strings.Join([]string{ + stubBrokerBin, + "--cluster-manager-path", clusterMgrBin, + "--config-dir", filepath.Join(base, "cluster"), + "--port", strconv.Itoa(node.port), + }, " ") + // nvpair-tui resolves its broker next to itself unless told otherwise, and + // --broker-path takes a single executable, so the fixture's arguments ride + // in a one-line wrapper script. + wrapper := filepath.Join(base, "broker.sh") + if err := os.WriteFile(wrapper, []byte("#!/bin/sh\nexec "+brokerArgs+" \"$@\"\n"), 0o700); err != nil { + t.Fatalf("write the broker wrapper: %v", err) + } + + node.cmd = ptyCommand(t, []string{ + tuiBin, + "--broker-path", wrapper, + "--control-socket", node.socket, + "--log-level", "debug", + }) + // A fresh HOME and friends: the per-user data directory on a developer + // machine belongs to whatever PAIR is already installed there. + node.cmd.Env = append(os.Environ(), + "HOME="+base, "XDG_CONFIG_HOME="+base, "XDG_RUNTIME_DIR="+base, + "APPDATA="+base, "LOCALAPPDATA="+base, + ) + // Its own process group, so cleanup can take the pty helper and the TUI + // down together. Killing `script` alone leaves the TUI running, holding + // its cluster manager and its socket into the next test. + node.cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + + logFile, err := os.Create(filepath.Join(base, name+".term")) + if err != nil { + t.Fatalf("create the terminal capture: %v", err) + } + node.log = logFile + node.cmd.Stdout = logFile + node.cmd.Stderr = logFile + + stdin, err := node.cmd.StdinPipe() + if err != nil { + t.Fatalf("pty stdin: %v", err) + } + node.stdin = stdin + if err := node.cmd.Start(); err != nil { + t.Fatalf("start nvpair-tui under a pty: %v", err) + } + t.Cleanup(node.stop) + + node.awaitControlSocket() + return node +} + +// stop quits the TUI the way a user does, then makes sure nothing survives. +func (n *tuiNode) stop() { + // `q` is the TUI's quit key; quitting shuts the broker down cleanly, which + // takes the cluster manager with it. + _, _ = n.stdin.Write([]byte("q")) + done := make(chan struct{}) + go func() { _ = n.cmd.Wait(); close(done) }() + select { + case <-done: + case <-time.After(25 * time.Second): + } + if n.cmd.Process != nil { + // The whole group: `script`, the TUI, its broker and the manager. + _ = syscall.Kill(-n.cmd.Process.Pid, syscall.SIGKILL) + } + if n.t.Failed() && n.log != nil { + if captured, err := os.ReadFile(n.log.Name()); err == nil { + n.t.Logf("%s terminal output:\n%s", n.name, strings.ReplaceAll(string(captured), "\r", "\n")) + } + } + _ = n.log.Close() +} + +// awaitControlSocket blocks until the node's socket answers a ping with a +// ready broker, so nothing is asked of a TUI that is still starting. +func (n *tuiNode) awaitControlSocket() { + n.t.Helper() + deadline := time.Now().Add(60 * time.Second) + var last string + for time.Now().Before(deadline) { + out, err := n.run("pending", "--json") + if err == nil && strings.Contains(out, "invites") { + return + } + last = out + time.Sleep(250 * time.Millisecond) + } + n.t.Fatalf("%s never served its control socket at %s; last answer = %q", n.name, n.socket, last) +} + +// run invokes one nvpair-tui subcommand against this node and returns its +// combined output. A non-zero exit comes back as an error carrying the code. +func (n *tuiNode) run(args ...string) (string, error) { + return n.runWithEnv(nil, args...) +} + +func (n *tuiNode) runWithEnv(extraEnv []string, args ...string) (string, error) { + cmd := exec.Command(tuiBin, append(args, "--control-socket", n.socket)...) + cmd.Env = append(os.Environ(), extraEnv...) + out, err := cmd.CombinedOutput() + return string(out), err +} + +// mustRun fails the test when the subcommand exits non-zero. +func (n *tuiNode) mustRun(args ...string) string { + n.t.Helper() + out, err := n.run(args...) + if err != nil { + n.t.Fatalf("%s: nvpair-tui %s failed (%v):\n%s", n.name, strings.Join(args, " "), err, out) + } + return out +} + +// exitCode digs the process exit status out of a run error. +func exitCode(t *testing.T, err error) int { + t.Helper() + if err == nil { + return 0 + } + var exitErr *exec.ExitError + if !asExitError(err, &exitErr) { + t.Fatalf("expected a process exit status, got %T (%v)", err, err) + } + return exitErr.ExitCode() +} + +func asExitError(err error, target **exec.ExitError) bool { + if e, ok := err.(*exec.ExitError); ok { + *target = e + return true + } + return false +} + +func TestHeadlessPairingOverTheControlSocket(t *testing.T) { + inviter := startTUINode(t, "inviter") + joiner := startTUINode(t, "joiner") + + t.Run("nothing is pending on a machine nobody has invited", func(t *testing.T) { + out := inviter.mustRun("pending", "--json") + if strings.TrimSpace(out) != `{"invites":[]}` { + t.Errorf("pending = %q, want an empty list", out) + } + }) + + t.Run("an unclustered machine reports no cluster", func(t *testing.T) { + var membership struct { + ClusterID string `json:"clusterId"` + Members []map[string]any `json:"members"` + } + if err := json.Unmarshal([]byte(inviter.mustRun("members", "--json")), &membership); err != nil { + t.Fatalf("decode members: %v", err) + } + if membership.ClusterID != "" { + t.Errorf("clusterId = %q, want empty before any pairing", membership.ClusterID) + } + }) + + // The invite. Its PIN is what an operator reads to the other machine. + var invite struct { + InviteID string `json:"inviteId"` + State string `json:"state"` + Pin string `json:"pin"` + } + t.Run("invite prints a PIN", func(t *testing.T) { + raw := inviter.mustRun("invite", "127.0.0.1", "--port", strconv.Itoa(joiner.port), "--json") + if err := json.Unmarshal([]byte(strings.TrimSpace(raw)), &invite); err != nil { + t.Fatalf("decode invite (%q): %v", raw, err) + } + if invite.State != "pending" { + t.Fatalf("state = %q, want pending", invite.State) + } + if len(invite.Pin) != 6 { + t.Fatalf("pin = %q, want six digits", invite.Pin) + } + + // The human form has to carry the PIN too — it is the whole point of + // the command. + human, err := inviter.run("invite", "127.0.0.1", "--port", strconv.Itoa(joiner.port)) + if err == nil && !strings.Contains(human, "PIN") { + t.Errorf("the human invite line does not show a PIN:\n%s", human) + } + }) + + t.Run("the invitation is waiting on the other machine", func(t *testing.T) { + out := awaitPending(t, joiner) + if !strings.Contains(out, "inviteId") { + t.Fatalf("pending = %q, want the inbound invite", out) + } + if strings.Contains(out, invite.Pin) { + t.Errorf("pair:pending disclosed the PIN:\n%s", out) + } + // The human listing names the inviter and the invite. + listed := joiner.mustRun("pending") + if !strings.Contains(listed, "ago") { + t.Errorf("the pending listing has no age column:\n%s", listed) + } + }) + + t.Run("a wrong PIN is refused with exit 1", func(t *testing.T) { + out, err := joiner.runWithEnv([]string{"NVPAIR_PIN=000000"}, "accept") + if code := exitCode(t, err); code != 1 { + t.Fatalf("exit = %d, want 1 for a wrong PIN:\n%s", code, out) + } + if strings.Contains(out, "000000") { + t.Errorf("the rejected PIN was echoed back:\n%s", out) + } + }) + + t.Run("the right PIN pairs the two machines", func(t *testing.T) { + // A wrong PIN terminates that invite on both sides, so the inviter + // sends a fresh one — which is what an operator does too. + raw := inviter.mustRun("invite", "127.0.0.1", "--port", strconv.Itoa(joiner.port), "--json") + if err := json.Unmarshal([]byte(strings.TrimSpace(raw)), &invite); err != nil { + t.Fatalf("decode the replacement invite: %v", err) + } + awaitPending(t, joiner) + + // The PIN travels in the environment, which is the form a script uses: + // an argument would be visible in ps to every user on the machine. + out, err := joiner.runWithEnv([]string{"NVPAIR_PIN=" + invite.Pin}, "accept") + if err != nil { + t.Fatalf("accept failed (%v):\n%s", err, out) + } + if !strings.Contains(out, "paired") { + t.Errorf("accept said %q, want it to report the pairing", out) + } + }) + + t.Run("both machines list the pair", func(t *testing.T) { + inviterCluster := awaitClustered(t, inviter) + joinerCluster := awaitClustered(t, joiner) + if inviterCluster != joinerCluster { + t.Errorf("the two machines joined different clusters: %q vs %q", inviterCluster, joinerCluster) + } + for _, node := range []*tuiNode{inviter, joiner} { + listed := node.mustRun("members") + if !strings.Contains(listed, inviterCluster) { + t.Errorf("%s members does not name the cluster:\n%s", node.name, listed) + } + if strings.Count(strings.TrimSpace(listed), "\n") < 2 { + t.Errorf("%s members lists fewer than two machines:\n%s", node.name, listed) + } + } + }) + + t.Run("the answered invite is no longer pending", func(t *testing.T) { + out := joiner.mustRun("pending", "--json") + if strings.TrimSpace(out) != `{"invites":[]}` { + t.Errorf("pending = %q, want the answered invite gone", out) + } + }) + + t.Run("accepting with nothing pending exits 2", func(t *testing.T) { + out, err := joiner.runWithEnv([]string{"NVPAIR_PIN=123456"}, "accept") + if code := exitCode(t, err); code != 2 { + t.Fatalf("exit = %d, want 2:\n%s", code, out) + } + if !strings.Contains(out, "no invite is pending") { + t.Errorf("output = %q, want it to say nothing is waiting", out) + } + }) + + t.Run("a subcommand with no running instance exits 2", func(t *testing.T) { + cmd := exec.Command(tuiBin, "members", "--control-socket", filepath.Join(shortDir(t, "empty"), "tui.sock")) + out, err := cmd.CombinedOutput() + if code := exitCode(t, err); code != 2 { + t.Fatalf("exit = %d, want 2:\n%s", code, out) + } + if !strings.Contains(string(out), "no nvpair-tui is listening") { + t.Errorf("output = %q, want it to say nothing is listening", out) + } + }) +} + +func TestDeclineFromTheControlSocket(t *testing.T) { + inviter := startTUINode(t, "decl-inviter") + joiner := startTUINode(t, "decl-joiner") + + inviter.mustRun("invite", "127.0.0.1", "--port", strconv.Itoa(joiner.port), "--json") + awaitPending(t, joiner) + + out := joiner.mustRun("decline") + if !strings.Contains(out, "declined") { + t.Errorf("decline said %q, want it to report the refusal", out) + } + if pending := joiner.mustRun("pending", "--json"); strings.TrimSpace(pending) != `{"invites":[]}` { + t.Errorf("pending = %q, want the declined invite gone", pending) + } +} + +// awaitPending blocks until an invitation is waiting on the node, and returns +// the raw pair:pending answer. +func awaitPending(t *testing.T, node *tuiNode) string { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + var last string + for time.Now().Before(deadline) { + last = node.mustRun("pending", "--json") + if !strings.Contains(last, `"invites":[]`) { + return last + } + time.Sleep(250 * time.Millisecond) + } + t.Fatalf("%s never saw an inbound invitation; last pending = %q", node.name, last) + return "" +} + +// awaitClustered blocks until the node reports a cluster id and returns it. +// Membership is durable the moment the pairing completes, but each side +// records it on its own schedule. +func awaitClustered(t *testing.T, node *tuiNode) string { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + var last string + for time.Now().Before(deadline) { + last = node.mustRun("members", "--json") + var membership struct { + ClusterID string `json:"clusterId"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(last)), &membership); err == nil && membership.ClusterID != "" { + return membership.ClusterID + } + time.Sleep(250 * time.Millisecond) + } + t.Fatalf("%s never reported a cluster; last members = %q", node.name, last) + return "" +} diff --git a/services/tests/main_test.go b/services/tests/main_test.go index c5d63c9c..3d95adbd 100644 --- a/services/tests/main_test.go +++ b/services/tests/main_test.go @@ -31,6 +31,8 @@ var ( manualNodesBin string clusterMgrBin string schedulerBin string + tuiBin string + stubBrokerBin string ) func TestMain(m *testing.M) { @@ -56,6 +58,8 @@ func TestMain(m *testing.M) { manualNodesBin = filepath.Join(tmpDir, "nvpair-manual-nodes"+ext) clusterMgrBin = filepath.Join(tmpDir, "nvpair-cluster-manager"+ext) schedulerBin = filepath.Join(tmpDir, "nvpair-job-scheduler"+ext) + tuiBin = filepath.Join(tmpDir, "nvpair-tui"+ext) + stubBrokerBin = filepath.Join(tmpDir, "stubbroker"+ext) log.Println("building ollama-proxy...") if err := goBuild(filepath.Join("..", "ollama-proxy"), proxyBin); err != nil { @@ -143,6 +147,22 @@ func TestMain(m *testing.M) { log.Fatalf("build nvpair-job-scheduler: %v", err) } + // The headless-pairing test drives the real nvpair-tui binary and its + // subcommands. Its broker is the stubbroker fixture rather than the real + // one, because every broker-owned port is a compiled-in constant and that + // test needs two independent nodes on one machine. + log.Println("building nvpair-tui...") + if err := goBuild(filepath.Join("..", "nvpair-tui"), tuiBin); err != nil { + os.RemoveAll(tmpDir) + log.Fatalf("build nvpair-tui: %v", err) + } + + log.Println("building the stubbroker fixture...") + if err := goBuild(filepath.Join("fixtures", "stubbroker"), stubBrokerBin); err != nil { + os.RemoveAll(tmpDir) + log.Fatalf("build stubbroker: %v", err) + } + code := m.Run() os.RemoveAll(tmpDir) os.Exit(code) diff --git a/services/versions.json b/services/versions.json index 29d8c230..3609cd67 100644 --- a/services/versions.json +++ b/services/versions.json @@ -1,7 +1,7 @@ { "$comment": "Single source of truth for all version numbers. See VERSIONING.md for bump rules.", - "product": "0.91.7", - "installer": "0.91.7", + "product": "0.92.0", + "installer": "0.92.0", "components": { "ollama-proxy": "0.26.2", "lmstudio-proxy": "0.16.2", @@ -15,6 +15,6 @@ "nvpair-engine-manager": "0.17.4", "nvpair-cluster-manager": "1.1.4", "nvpair-job-scheduler": "0.4.1", - "nvpair-tui": "0.7.2" + "nvpair-tui": "0.8.0" } }