Skip to content

Adopt externally-managed OpenAI-compatible endpoints by base URL - #64

Open
vonargo wants to merge 7 commits into
NVIDIA:mainfrom
vonargo:feat/openai-endpoint
Open

vonargo wants to merge 7 commits into
NVIDIA:mainfrom
vonargo:feat/openai-endpoint

Conversation

@vonargo

@vonargo vonargo commented Sep 11, 2026

Copy link
Copy Markdown

Adopt externally-managed OpenAI-compatible endpoints by base URL

Part of #24 (minimal slice). This adds a way to adopt a server that already speaks the
OpenAI HTTP API — vLLM, llama.cpp server, TGI, or anything similar — by declaring its
base URL, without PAIR needing to know which software sits behind it. vLLM is the test
fixture, not the feature.

What changed

  • nvpair-manual-nodes (0.11.1 → 0.12.0)node/add accepts exactly one of
    address (historical form: host probed on the default engine ports) or
    openai_base_url (http only in this release). Declared endpoints are probed at
    their own URL: GET {base}/models doubles as liveness + model list
    (openai_up / openai_models), plus best-effort node-info on the URL's host.
    The default engine-port legs are not probed for this entry type. The service now
    owns its entry list in the app data directory (manual-nodes.json, atomic
    tmp+rename) and restores it at startup, so manual nodes survive a restart.
  • nvpair-ui-broker (0.40.2 → 0.41.0) — bridges declared endpoints into
    lmstudio-proxy with their path prefix; attributes openai_models to the new
    "openai" engine in modelsByEngine; translates node/remove by UUID back to
    the manual store key for the endpoint leg. An endpoint keeps its manual key
    (the declared URL's identity) and never adopts the host's learned node-info
    UUID: an endpoint on the same box as PAIR — or two endpoints on one host —
    must not fold into that machine's own node, where they would clobber each
    other on every probe.
  • lmstudio-proxy (0.16.2 → 0.17.0) — manual nodes may carry base_path
    (e.g. /v1). The /v1 prefix is the proxy's own API root, so inbound
    /v1/... paths are joined onto the endpoint's root for forwarding and model-list
    fanout; nodes without a base path forward verbatim as before. Workloads served by
    a base-prefixed endpoint are labeled engine: "openai" (including on failover).
  • nvpair-tui (0.7.2 → 0.9.0) — the add prompt accepts a bare host or
    http://host:port/v1; the manual table gains an OPENAI reachability column.
    Also fixes a pre-existing table overflow found while testing: the upstream
    table styling rendered rows wider than the widths the existing tab layouts
    budgeted, and the table's viewport truncates rows at the terminal edge. The
    problem was mostly hidden because the truncated region often contained
    trailing whitespace; the added OpenAI column moved real content into the
    clip zone and made the clipping visible. Cells now render at their exact
    budgeted widths.
  • desktop — the Add-node dialog gains an "OpenAI endpoint" row (input + Add)
    that invokes the new nodes:add-endpoint ws channel, which relays
    node/add { openai_base_url } and returns inline errors. One-line mapping of
    workload engine openai onto the existing lm-studio display path so
    external-endpoint jobs are not dropped by the closed engine union.
  • stability (found while dogfooding) — same-box endpoints were
    re-bridged into the proxies (remove + add) on every 10s probe: the
    prober's change diff counted msSince (the GPU sample's age, which
    advances every probe by construction), so every node-info-up manual
    node emitted a state change each cycle, and the broker re-issued the
    bridge unconditionally. msSince is now excluded from the diff
    (display-only — the store stays fresh for nodes/list), and the
    manual→proxy bridge skips the RPC when the exact intent (add payload
    or remove) was already applied to that proxy + engine + key slot.
  • docs + versions — manual-nodes and lmstudio-proxy READMEs,
    docs/architecture.mdx manual-node paragraph, services/versions.json
    (product/installer 0.93.0).

Wire contract

  • node/add param + status echo: openai_base_url
  • manual-nodes status: openai_up, openai_host, openai_port, openai_base_path, openai_models
  • proxy Node / node/add-manual: base_path
  • workload engine / modelsByEngine key: "openai"
  • desktop ws channel: nodes:add-endpoint { request: { url }; response: { ok, error? } }
  • entry rule: exactly one of address | openai_base_url; http only (v1)
  • identity rule: address entries re-key to the host's learned UUID (dedup with
    mDNS, as before); endpoint entries always keep their manual id

Verification

  • Per-module Go suites green (fresh runs): nvpair-manual-nodes, nvpair-ui-broker,
    lmstudio-proxy (incl. a new real-binary e2e: models fanout + inference land on
    the endpoint's /v1-rooted paths without /v1/v1 doubling, workload labeled
    engine: "openai"), nvpair-tui; cross-process broker suite (bridge tests incl.
    the new endpoint bridge) green.
  • Desktop: typecheck, test:unit (213 passing), service-contracts:check,
    dead-code:check, lint all clean.
  • TUI width-sweep test: the manual table's last column survives the viewport
    truncation at widths {44, 60, 80, 120, 168} — regression guard for the
    overflow fix above.
  • Churn regression guards: nvpair-manual-nodes (a probe that changes only
    the telemetry sample age emits no node/updated) and nvpair-ui-broker
    (bridge-intent equality: identical add payload / remove is equal, any
    routing-relevant difference is not).
  • Service-level walkthrough (live vLLM at 127.0.0.1:8888):
    1. node/add {"openai_base_url":"http://127.0.0.1:8888/v1"} → initial unprobed
      echo with parsed host/port/base path; then node/discovered with
      openai_up: true and the endpoint's model inventory.
    2. lmstudio-proxy --port 18099 + node/add-manual {..., "base_path":"/v1"}
      GET /v1/models on the proxy returns the endpoint's inventory.
    3. POST /v1/chat/completions on the proxy → routed to the endpoint (proxy log:
      node_id: "stub", target: "127.0.0.1:8888", 200), real completion returned.
    4. Kill the manager, restart, nodes/list → the entry comes back from
      manual-nodes.json (persisted + restored).

Known limitations (follow-ups, out of scope here)

  • Endpoint nodes render with the "LM Studio" engine chip on desktop node cards;
    honest per-engine display is part of [Feature]: Adopt arbitrary OpenAI-compatible backends with endpoint manifests, model aliases, and deterministic capability routing #24's broader UI work.
  • https endpoints are rejected (v1 is http only); no auth headers.
  • Endpoints are not advertised to other peers' proxies (cluster-wide reachability
    of manual nodes is Make a node added by address a first-class peer (overlay networks such as Tailscale) #10 territory).
  • The desktop's legacy manual-nodes.json store + replay remain in place
    (untouched; nothing on main wrote entries there). Removal is a follow-up cleanup.
  • The desktop has an add path but no remove path for endpoint entries:
    node removal in the desktop is cluster-scoped, and endpoints are not cluster
    members. Remove one in the TUI Manual view (r on the selected row) or by
    editing the service store. A desktop-side affordance is follow-up UI work.
  • No name field in the v1 add flows — the node ID defaults to
    manual:<host>:<port>; the service still accepts a name param for future UIs.
  • Residual steady-state log lines (pre-existing, unchanged here): manual
    nodes still emit node/updated each probe cycle because that event also
    carries the node's telemetry to the scheduler (gating it would make a
    healthy endpoint age out as stale within the scheduler's 10s freshness
    window). The scheduler's periodic priority reconciliation and the
    proxies' local-backend refresh therefore still log on a regular cadence;
    the redundant bridge remove/add churn is gone. Tightening the scheduler's
    freshness window vs. probe cadence, and log verbosity in
    set-local-backend handling, are follow-ups.

…entries

node/add accepts exactly one of address (historical default-port probing)
or openai_base_url (http only). Declared endpoints are probed at their
own URL: GET {base}/models doubles as liveness and model list, with
best-effort node-info on the URL's host; the default engine-port legs
are skipped. The service now owns its entry list in the app data
directory (atomic tmp+rename) and restores it at startup, so manual
nodes survive a restart.

Signed-off-by: Will Ford <will@vonargo.ai>
Declared endpoints flow through to the proxy carrying their path
prefix; their model lists attribute to the new 'openai' engine in
modelsByEngine, and node/remove by UUID translates back to the manual
store key for the endpoint leg. Endpoint entries keep their manual
identity (the declared URL) and never adopt the host's learned
node-info UUID, so endpoints on the same box as PAIR stay separate
nodes instead of clobbering each other.

Signed-off-by: Will Ford <will@vonargo.ai>
External OpenAI-compatible endpoints adopted via a declared base URL
(e.g. http://dgx:8000/v1) carry a base_path into the proxy. The /v1
prefix is the proxy's own API root, so inbound /v1/... paths are
joined onto the endpoint's root instead of forwarded verbatim: model
list fanout, the reverse-proxy Director, and the workload engine
label (openai) all follow the endpoint's path. Nodes without a base
path (discovered peers, classic manual nodes) are unchanged.

Signed-off-by: Will Ford <will@vonargo.ai>
The TUI add prompt accepts a full http:// base URL in addition to a
bare host (with an OPENAI reachability column). The desktop Add-node
dialog gains an 'OpenAI endpoint' row that relays node/add with
openai_base_url (nvpair-manual-nodes persists the entry). Workload
engine 'openai' maps onto the existing lm-studio display path so
external-endpoint jobs are not dropped by the closed union.

Signed-off-by: Will Ford <will@vonargo.ai>
Document the openai_base_url entry flavor, service-owned entry
persistence, the proxy's base_path forwarding, and the endpoint
identity rule. Bump manual-nodes 0.12.0, ui-broker 0.41.0,
lmstudio-proxy 0.17.0, tui 0.8.0, product 0.92.0.

Signed-off-by: Will Ford <will@vonargo.ai>
bubbles' default table styles add one space of padding to each side of
every cell, so rows rendered 2*n_columns wider than the width the views
budget, and the table viewport's hard truncation cut the excess off the
right edge — clipping the rightmost column of every tab at any terminal
width (first visibly so on the Manual table's fifth column). Drop the
default cell/header padding so rows render at their exact budgeted width,
and add a width-sweep test asserting the last column survives.

Signed-off-by: Will Ford <will@vonargo.ai>
The prober change diff included msSince, which advances on every probe. That meant any manual node with node-info available emitted a state change every probe cycle, even when nothing related to routing had changed. Same-box endpoints hit this consistently.
Each update re-entered the manual-to-proxy bridge and reissued the add or remove RPC. Re-adding a manual node removes and reinserts it on the proxy side, causing log churn and unnecessary priority snapshot rebuilds.
msSince is only used for display. The prober store already keeps the current value for nodes/list, so remove it from change detection.
Also make the bridge idempotent by skipping the RPC when the same add or remove intent has already been applied to that proxy, engine, and key. Explicit removes and prober crashes clear the record so a later add is bridged again. Respawned proxies are reseeded automatically.

Signed-off-by: Will Ford <will@vonargo.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant