Make proxy upstream response-header timeout configurable

README.md

@@ -239,6 +239,8 @@ Each entry assumes the ones before it.
 8. **[Developer guide](docs/developing.mdx)** — read this before contributing:
    where the code lives, how a change travels through the layers, and the
    conventions the project enforces.
+9. **[Proxy response-header timeout](docs/proxy-response-header-timeout.mdx)** —
+   the 120 s upstream header deadline, when it trips, and how to configure it.
 
 Component references, for when you already know what you are looking for:
 

docs/proxy-response-header-timeout.mdx

@@ -0,0 +1,69 @@
+{/*
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+*/}
+
+# Proxy response-header timeout
+
+Both inference proxies (`ollama-proxy`, `lmstudio-proxy`) cap how long they
+wait for the upstream engine to send response headers: `ResponseHeaderTimeout`
+on the upstream transport. The default is 120 s, and it is now configurable.
+
+## Why the timeout exists, and why 120 s is not always enough
+
+An engine sends no response headers until generation starts. Two normal
+situations delay that past 120 s:
+
+- **Queueing.** Ollama serves one request at a time by default
+  (`OLLAMA_NUM_PARALLEL=1`). Twenty concurrent requests at ~11 s each means
+  the last one waits ~220 s for its turn — every request past the 120 s mark
+  fails through PAIR with a 502 while succeeding direct.
+- **Cold model loads.** The first request after a model change waits on the
+  load before any header is sent.
+
+When the timeout trips, the proxy closes the upstream connection (which also
+cancels the engine's queued work) and answers
+`502 {"error":"upstream error: net/http: timeout awaiting response headers"}`.
+
+```mermaid
+sequenceDiagram
+    participant App
+    participant Proxy
+    participant Engine
+
+    App->>Proxy: POST /v1/chat/completions
+    Proxy->>Engine: forward (ResponseHeaderTimeout = T)
+    alt headers within T
+        Engine-->>Proxy: headers, then stream
+        Proxy-->>App: 200 stream
+    else no headers within T
+        Note over Proxy: close upstream,<br/>cancel engine work
+        Proxy-->>App: 502 timeout awaiting response headers
+    end
+```
+
+## Configuring it
+
+Precedence is flag, then environment, then the 120 s default:
+
+```bash
+ollama-proxy --response-header-timeout 10m
+NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT=10m ollama-proxy
+```
+
+The value is a Go duration (`30s`, `5m`, `1h30m`). The broker spawns both
+proxies as child processes, so the environment variable set on the broker (or
+the desktop app / headless launcher) is inherited — no broker change is
+needed. A missing, unparseable, or non-positive value logs a warning and
+falls back to 120 s, so a bad setting can never silently disable the timeout.
+
+The effective value is logged at proxy startup (`response_header_timeout`)
+for post-mortem analysis.
+
+## Choosing a value
+
+Match it to the worst case you want to survive: queue depth × per-request
+time, or the slowest cold load on your hardware. There is no correctness cost
+to a generous value — it only bounds how long a request waits on an engine
+that never answers. Keep it shorter than any client-side timeout above PAIR
+so failures still surface where you expect them.

docs/troubleshooting.mdx

@@ -116,6 +116,19 @@ PAIR's port, expand **Engine settings > Ports** on the node's card in
 PAIR's port is usually easier and leaves the other application alone. Refer to
 [Changing a Port](getting-started.mdx#changing-a-port).
 
+## Inference Requests Fail with 502 After 120 Seconds
+
+`502 {"error":"upstream error: net/http: timeout awaiting response headers"}`
+means the engine did not send response headers within the proxy's upstream
+response-header timeout (default 120 s). Engines send no headers until
+generation starts, so this trips on requests queued behind other work
+(Ollama serves one request at a time by default) or on slow model loads —
+while the same request sent directly to the engine succeeds. Raise the
+timeout with the `--response-header-timeout` flag (a Go duration, e.g. `5m`)
+or the `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT` environment variable on
+`ollama-proxy` / `lmstudio-proxy`. Refer to
+[Proxy response-header timeout](proxy-response-header-timeout.mdx).
+
 ## Requests Work but PAIR Shows No Jobs
 
 If inference succeeds and yet **Jobs** stays empty, and the machine you sent the

services/lmstudio-proxy/README.md

@@ -30,6 +30,7 @@ lmstudio-proxy [flags]
 | `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |
 | `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |
 | `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |
+| `--response-header-timeout` | *(`$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s`)* | Upstream response-header timeout (Go duration, e.g. `5m`). A request whose engine has not sent response headers within this long fails with 502. Raise it for deep engine queues or slow model loads. |
 | `--version` | | Print version and exit |
 
 ### HTTP Reverse Proxy

services/lmstudio-proxy/header_timeout_test.go

@@ -0,0 +1,45 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+package main
+
+import (
+	"testing"
+	"time"
+)
+
+func TestResolveResponseHeaderTimeout(t *testing.T) {
+	cases := []struct {
+		name string
+		flag string
+		env  string
+		want time.Duration
+	}{
+		{"default", "", "", 120 * time.Second},
+		{"env", "", "5m", 5 * time.Minute},
+		{"flag beats env", "30s", "5m", 30 * time.Second},
+		{"invalid env falls back", "", "bogus", 120 * time.Second},
+		{"zero env falls back", "", "0", 120 * time.Second},
+		{"negative flag falls back", "-1s", "", 120 * time.Second},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Setenv(responseHeaderTimeoutEnv, tc.env)
+			if got := resolveResponseHeaderTimeout(tc.flag); got != tc.want {
+				t.Fatalf("resolveResponseHeaderTimeout(%q) = %s, want %s", tc.flag, got, tc.want)
+			}
+		})
+	}
+}
+
+// The configured timeout must reach the upstream transports built after
+// startup.
+func TestProxyTransportUsesConfiguredHeaderTimeout(t *testing.T) {
+	old := proxyResponseTimeout
+	defer func() { proxyResponseTimeout = old }()
+	proxyResponseTimeout = 10 * time.Minute
+	tr := newProxyTransport(nil)
+	if tr.ResponseHeaderTimeout != 10*time.Minute {
+		t.Fatalf("ResponseHeaderTimeout = %s, want 10m", tr.ResponseHeaderTimeout)
+	}
+}

services/lmstudio-proxy/main.go

@@ -23,10 +23,13 @@ func main() {
 	ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists")
 	ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)")
 	clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered")
+	responseHeaderTimeout := flag.String("response-header-timeout", "", "upstream response header timeout (Go duration, e.g. 5m); default: $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT or 120s")
 	showVersion := flag.Bool("version", false, "print version and exit")
 	resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo)
 	flag.Parse()
 
+	proxyResponseTimeout = resolveResponseHeaderTimeout(*responseHeaderTimeout)
+
 	if *showVersion {
 		fmt.Println(Version)
 		os.Exit(0)

services/lmstudio-proxy/proxy.go

@@ -19,6 +19,7 @@ import (
 	"net/http"
 	"net/http/httputil"
 	"net/url"
+	"os"
 	"sort"
 	"strconv"
 	"sync"
@@ -510,18 +511,55 @@ func (p *Proxy) Run(ctx context.Context) error {
 // Timeouts for upstream connections. Logged at startup so they're always
 // present in any captured log for post-mortem analysis.
 const (
-	proxyDialTimeout     = 10 * time.Second
-	proxyKeepAlive       = 30 * time.Second
-	proxyResponseTimeout = 120 * time.Second
-	proxyMaxIdleConns    = 50
-	proxyIdleConnTimeout = 90 * time.Second
+	proxyDialTimeout = 10 * time.Second
+	proxyKeepAlive   = 30 * time.Second
+	// defaultProxyResponseTimeout is the upstream response-header timeout
+	// unless --response-header-timeout (or NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT)
+	// overrides it. The engine sends no headers until generation starts, so a
+	// request queued behind other work, or waiting on a slow model load, needs
+	// longer than this to survive the proxy.
+	defaultProxyResponseTimeout = 120 * time.Second
+	proxyMaxIdleConns           = 50
+	proxyIdleConnTimeout        = 90 * time.Second
 	// Inbound http.Server limits — keep IdleTimeout aligned with client
 	// IdleConnTimeout so idle keep-alives are reaped on both sides.
 	proxyReadHeaderTimeout = 10 * time.Second
 	proxyServerIdleTimeout = 90 * time.Second
 	maxModelListBytes      = 16 << 20
 )
 
+// responseHeaderTimeoutEnv carries the upstream response-header timeout when
+// the --response-header-timeout flag is empty. The broker spawns the proxy as
+// a child process, so the variable set on the broker (or desktop) is inherited
+// without any broker change.
+const responseHeaderTimeoutEnv = "NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT"
+
+// proxyResponseTimeout is the effective upstream response-header timeout,
+// resolved at startup: --response-header-timeout flag, then
+// NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then the 120 s default. Upstream
+// transports are built lazily via newProxyTransport, so assigning it before
+// serving is sufficient.
+var proxyResponseTimeout = defaultProxyResponseTimeout
+
+// resolveResponseHeaderTimeout applies the flag > env > default precedence. A
+// missing, unparseable, or non-positive value falls back to the default, so a
+// bad setting can never silently disable the timeout.
+func resolveResponseHeaderTimeout(flagVal string) time.Duration {
+	raw := flagVal
+	if raw == "" {
+		raw = os.Getenv(responseHeaderTimeoutEnv)
+	}
+	if raw == "" {
+		return defaultProxyResponseTimeout
+	}
+	d, err := time.ParseDuration(raw)
+	if err != nil || d <= 0 {
+		log.Printf("invalid response-header timeout %q, using default %s", raw, defaultProxyResponseTimeout)
+		return defaultProxyResponseTimeout
+	}
+	return d
+}
+
 // idleClientWriteTimeout bounds how long a single write of streamed response
 // bytes to the client may block. A killed client can leave a half-open socket
 // whose kernel send buffer fills and never drains; without this deadline the

services/ollama-proxy/README.md

@@ -29,6 +29,7 @@ ollama-proxy [flags]
 | `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC |
 | `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. |
 | `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. |
+| `--response-header-timeout` | *(`$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s`)* | Upstream response-header timeout (Go duration, e.g. `5m`). A request whose engine has not sent response headers within this long fails with 502. Raise it for deep engine queues or slow model loads. |
 | `--version` | | Print version and exit |
 
 ### HTTP Reverse Proxy

services/ollama-proxy/header_timeout_test.go

@@ -0,0 +1,45 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+package main
+
+import (
+	"testing"
+	"time"
+)
+
+func TestResolveResponseHeaderTimeout(t *testing.T) {
+	cases := []struct {
+		name string
+		flag string
+		env  string
+		want time.Duration
+	}{
+		{"default", "", "", 120 * time.Second},
+		{"env", "", "5m", 5 * time.Minute},
+		{"flag beats env", "30s", "5m", 30 * time.Second},
+		{"invalid env falls back", "", "bogus", 120 * time.Second},
+		{"zero env falls back", "", "0", 120 * time.Second},
+		{"negative flag falls back", "-1s", "", 120 * time.Second},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Setenv(responseHeaderTimeoutEnv, tc.env)
+			if got := resolveResponseHeaderTimeout(tc.flag); got != tc.want {
+				t.Fatalf("resolveResponseHeaderTimeout(%q) = %s, want %s", tc.flag, got, tc.want)
+			}
+		})
+	}
+}
+
+// The configured timeout must reach the upstream transports built after
+// startup.
+func TestProxyTransportUsesConfiguredHeaderTimeout(t *testing.T) {
+	old := proxyResponseTimeout
+	defer func() { proxyResponseTimeout = old }()
+	proxyResponseTimeout = 10 * time.Minute
+	tr := newProxyTransport(nil)
+	if tr.ResponseHeaderTimeout != 10*time.Minute {
+		t.Fatalf("ResponseHeaderTimeout = %s, want 10m", tr.ResponseHeaderTimeout)
+	}
+}

services/ollama-proxy/main.go

@@ -34,10 +34,13 @@ func main() {
 	ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists")
 	ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)")
 	clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered")
+	responseHeaderTimeout := flag.String("response-header-timeout", "", "upstream response header timeout (Go duration, e.g. 5m); default: $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT or 120s")
 	showVersion := flag.Bool("version", false, "print version and exit")
 	resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo)
 	flag.Parse()
 
+	proxyResponseTimeout = resolveResponseHeaderTimeout(*responseHeaderTimeout)
+
 	if *showVersion {
 		fmt.Println(Version)
 		os.Exit(0)

services/ollama-proxy/proxy.go

@@ -19,6 +19,7 @@ import (
 	"net/http"
 	"net/http/httputil"
 	"net/url"
+	"os"
 	"sort"
 	"strconv"
 	"strings"
@@ -526,18 +527,55 @@ func (p *Proxy) Run(ctx context.Context) error {
 // Timeouts for upstream connections. Logged at startup so they're always
 // present in any captured log for post-mortem analysis.
 const (
-	proxyDialTimeout     = 10 * time.Second
-	proxyKeepAlive       = 30 * time.Second
-	proxyResponseTimeout = 120 * time.Second
-	proxyMaxIdleConns    = 50
-	proxyIdleConnTimeout = 90 * time.Second
+	proxyDialTimeout = 10 * time.Second
+	proxyKeepAlive   = 30 * time.Second
+	// defaultProxyResponseTimeout is the upstream response-header timeout
+	// unless --response-header-timeout (or NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT)
+	// overrides it. Ollama sends no headers until generation starts, so a
+	// request queued behind other work, or waiting on a slow model load, needs
+	// longer than this to survive the proxy.
+	defaultProxyResponseTimeout = 120 * time.Second
+	proxyMaxIdleConns           = 50
+	proxyIdleConnTimeout        = 90 * time.Second
 	// Inbound http.Server limits — keep IdleTimeout aligned with client
 	// IdleConnTimeout so idle keep-alives are reaped on both sides.
 	proxyReadHeaderTimeout = 10 * time.Second
 	proxyServerIdleTimeout = 90 * time.Second
 	maxModelListBytes      = 16 << 20
 )
 
+// responseHeaderTimeoutEnv carries the upstream response-header timeout when
+// the --response-header-timeout flag is empty. The broker spawns the proxy as
+// a child process, so the variable set on the broker (or desktop) is inherited
+// without any broker change.
+const responseHeaderTimeoutEnv = "NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT"
+
+// proxyResponseTimeout is the effective upstream response-header timeout,
+// resolved at startup: --response-header-timeout flag, then
+// NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then the 120 s default. Upstream
+// transports are built lazily via newProxyTransport, so assigning it before
+// serving is sufficient.
+var proxyResponseTimeout = defaultProxyResponseTimeout
+
+// resolveResponseHeaderTimeout applies the flag > env > default precedence. A
+// missing, unparseable, or non-positive value falls back to the default, so a
+// bad setting can never silently disable the timeout.
+func resolveResponseHeaderTimeout(flagVal string) time.Duration {
+	raw := flagVal
+	if raw == "" {
+		raw = os.Getenv(responseHeaderTimeoutEnv)
+	}
+	if raw == "" {
+		return defaultProxyResponseTimeout
+	}
+	d, err := time.ParseDuration(raw)
+	if err != nil || d <= 0 {
+		log.Printf("invalid response-header timeout %q, using default %s", raw, defaultProxyResponseTimeout)
+		return defaultProxyResponseTimeout
+	}
+	return d
+}
+
 // idleClientWriteTimeout bounds how long a single write of streamed response
 // bytes to the client may block. A killed client can leave a half-open socket
 // whose kernel send buffer fills and never drains; without this deadline the

services/versions.json

@@ -3,8 +3,8 @@
   "product": "0.91.7",
   "installer": "0.91.7",
   "components": {
-    "ollama-proxy": "0.26.2",
-    "lmstudio-proxy": "0.16.2",
+    "ollama-proxy": "0.27.0",
+    "lmstudio-proxy": "0.17.0",
     "nvpair-node-info": "0.13.3",
     "nvpair-node-scanner": "0.20.3",
     "nvpair-manual-nodes": "0.11.1",