diff --git a/README.md b/README.md index 0f0a7242..1fbf1987 100644 --- a/README.md +++ b/README.md @@ -239,6 +239,8 @@ Each entry assumes the ones before it. 8. **[Developer guide](docs/developing.mdx)** — read this before contributing: where the code lives, how a change travels through the layers, and the conventions the project enforces. +9. **[Proxy response-header timeout](docs/proxy-response-header-timeout.mdx)** — + the 120 s upstream header deadline, when it trips, and how to configure it. Component references, for when you already know what you are looking for: diff --git a/docs/proxy-response-header-timeout.mdx b/docs/proxy-response-header-timeout.mdx new file mode 100644 index 00000000..47d901f6 --- /dev/null +++ b/docs/proxy-response-header-timeout.mdx @@ -0,0 +1,69 @@ +{/* +SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 +*/} + +# Proxy response-header timeout + +Both inference proxies (`ollama-proxy`, `lmstudio-proxy`) cap how long they +wait for the upstream engine to send response headers: `ResponseHeaderTimeout` +on the upstream transport. The default is 120 s, and it is now configurable. + +## Why the timeout exists, and why 120 s is not always enough + +An engine sends no response headers until generation starts. Two normal +situations delay that past 120 s: + +- **Queueing.** Ollama serves one request at a time by default + (`OLLAMA_NUM_PARALLEL=1`). Twenty concurrent requests at ~11 s each means + the last one waits ~220 s for its turn — every request past the 120 s mark + fails through PAIR with a 502 while succeeding direct. +- **Cold model loads.** The first request after a model change waits on the + load before any header is sent. + +When the timeout trips, the proxy closes the upstream connection (which also +cancels the engine's queued work) and answers +`502 {"error":"upstream error: net/http: timeout awaiting response headers"}`. + +```mermaid +sequenceDiagram + participant App + participant Proxy + participant Engine + + App->>Proxy: POST /v1/chat/completions + Proxy->>Engine: forward (ResponseHeaderTimeout = T) + alt headers within T + Engine-->>Proxy: headers, then stream + Proxy-->>App: 200 stream + else no headers within T + Note over Proxy: close upstream,
cancel engine work + Proxy-->>App: 502 timeout awaiting response headers + end +``` + +## Configuring it + +Precedence is flag, then environment, then the 120 s default: + +```bash +ollama-proxy --response-header-timeout 10m +NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT=10m ollama-proxy +``` + +The value is a Go duration (`30s`, `5m`, `1h30m`). The broker spawns both +proxies as child processes, so the environment variable set on the broker (or +the desktop app / headless launcher) is inherited — no broker change is +needed. A missing, unparseable, or non-positive value logs a warning and +falls back to 120 s, so a bad setting can never silently disable the timeout. + +The effective value is logged at proxy startup (`response_header_timeout`) +for post-mortem analysis. + +## Choosing a value + +Match it to the worst case you want to survive: queue depth × per-request +time, or the slowest cold load on your hardware. There is no correctness cost +to a generous value — it only bounds how long a request waits on an engine +that never answers. Keep it shorter than any client-side timeout above PAIR +so failures still surface where you expect them. diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index e4d8822c..3833c427 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -116,6 +116,19 @@ PAIR's port, expand **Engine settings > Ports** on the node's card in PAIR's port is usually easier and leaves the other application alone. Refer to [Changing a Port](getting-started.mdx#changing-a-port). +## Inference Requests Fail with 502 After 120 Seconds + +`502 {"error":"upstream error: net/http: timeout awaiting response headers"}` +means the engine did not send response headers within the proxy's upstream +response-header timeout (default 120 s). Engines send no headers until +generation starts, so this trips on requests queued behind other work +(Ollama serves one request at a time by default) or on slow model loads — +while the same request sent directly to the engine succeeds. Raise the +timeout with the `--response-header-timeout` flag (a Go duration, e.g. `5m`) +or the `NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT` environment variable on +`ollama-proxy` / `lmstudio-proxy`. Refer to +[Proxy response-header timeout](proxy-response-header-timeout.mdx). + ## Requests Work but PAIR Shows No Jobs If inference succeeds and yet **Jobs** stays empty, and the machine you sent the diff --git a/services/lmstudio-proxy/README.md b/services/lmstudio-proxy/README.md index 71a8b70d..c3825aa5 100644 --- a/services/lmstudio-proxy/README.md +++ b/services/lmstudio-proxy/README.md @@ -30,6 +30,7 @@ lmstudio-proxy [flags] | `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC | | `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. | | `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. | +| `--response-header-timeout` | *(`$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s`)* | Upstream response-header timeout (Go duration, e.g. `5m`). A request whose engine has not sent response headers within this long fails with 502. Raise it for deep engine queues or slow model loads. | | `--version` | | Print version and exit | ### HTTP Reverse Proxy diff --git a/services/lmstudio-proxy/header_timeout_test.go b/services/lmstudio-proxy/header_timeout_test.go new file mode 100644 index 00000000..0b38bfcb --- /dev/null +++ b/services/lmstudio-proxy/header_timeout_test.go @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "testing" + "time" +) + +func TestResolveResponseHeaderTimeout(t *testing.T) { + cases := []struct { + name string + flag string + env string + want time.Duration + }{ + {"default", "", "", 120 * time.Second}, + {"env", "", "5m", 5 * time.Minute}, + {"flag beats env", "30s", "5m", 30 * time.Second}, + {"invalid env falls back", "", "bogus", 120 * time.Second}, + {"zero env falls back", "", "0", 120 * time.Second}, + {"negative flag falls back", "-1s", "", 120 * time.Second}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Setenv(responseHeaderTimeoutEnv, tc.env) + if got := resolveResponseHeaderTimeout(tc.flag); got != tc.want { + t.Fatalf("resolveResponseHeaderTimeout(%q) = %s, want %s", tc.flag, got, tc.want) + } + }) + } +} + +// The configured timeout must reach the upstream transports built after +// startup. +func TestProxyTransportUsesConfiguredHeaderTimeout(t *testing.T) { + old := proxyResponseTimeout + defer func() { proxyResponseTimeout = old }() + proxyResponseTimeout = 10 * time.Minute + tr := newProxyTransport(nil) + if tr.ResponseHeaderTimeout != 10*time.Minute { + t.Fatalf("ResponseHeaderTimeout = %s, want 10m", tr.ResponseHeaderTimeout) + } +} diff --git a/services/lmstudio-proxy/main.go b/services/lmstudio-proxy/main.go index 846ac9cb..0467d769 100644 --- a/services/lmstudio-proxy/main.go +++ b/services/lmstudio-proxy/main.go @@ -23,10 +23,13 @@ func main() { ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists") ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)") clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered") + responseHeaderTimeout := flag.String("response-header-timeout", "", "upstream response header timeout (Go duration, e.g. 5m); default: $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT or 120s") showVersion := flag.Bool("version", false, "print version and exit") resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo) flag.Parse() + proxyResponseTimeout = resolveResponseHeaderTimeout(*responseHeaderTimeout) + if *showVersion { fmt.Println(Version) os.Exit(0) diff --git a/services/lmstudio-proxy/proxy.go b/services/lmstudio-proxy/proxy.go index 6e619e77..ef74db35 100644 --- a/services/lmstudio-proxy/proxy.go +++ b/services/lmstudio-proxy/proxy.go @@ -19,6 +19,7 @@ import ( "net/http" "net/http/httputil" "net/url" + "os" "sort" "strconv" "sync" @@ -510,11 +511,16 @@ func (p *Proxy) Run(ctx context.Context) error { // Timeouts for upstream connections. Logged at startup so they're always // present in any captured log for post-mortem analysis. const ( - proxyDialTimeout = 10 * time.Second - proxyKeepAlive = 30 * time.Second - proxyResponseTimeout = 120 * time.Second - proxyMaxIdleConns = 50 - proxyIdleConnTimeout = 90 * time.Second + proxyDialTimeout = 10 * time.Second + proxyKeepAlive = 30 * time.Second + // defaultProxyResponseTimeout is the upstream response-header timeout + // unless --response-header-timeout (or NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT) + // overrides it. The engine sends no headers until generation starts, so a + // request queued behind other work, or waiting on a slow model load, needs + // longer than this to survive the proxy. + defaultProxyResponseTimeout = 120 * time.Second + proxyMaxIdleConns = 50 + proxyIdleConnTimeout = 90 * time.Second // Inbound http.Server limits — keep IdleTimeout aligned with client // IdleConnTimeout so idle keep-alives are reaped on both sides. proxyReadHeaderTimeout = 10 * time.Second @@ -522,6 +528,38 @@ const ( maxModelListBytes = 16 << 20 ) +// responseHeaderTimeoutEnv carries the upstream response-header timeout when +// the --response-header-timeout flag is empty. The broker spawns the proxy as +// a child process, so the variable set on the broker (or desktop) is inherited +// without any broker change. +const responseHeaderTimeoutEnv = "NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT" + +// proxyResponseTimeout is the effective upstream response-header timeout, +// resolved at startup: --response-header-timeout flag, then +// NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then the 120 s default. Upstream +// transports are built lazily via newProxyTransport, so assigning it before +// serving is sufficient. +var proxyResponseTimeout = defaultProxyResponseTimeout + +// resolveResponseHeaderTimeout applies the flag > env > default precedence. A +// missing, unparseable, or non-positive value falls back to the default, so a +// bad setting can never silently disable the timeout. +func resolveResponseHeaderTimeout(flagVal string) time.Duration { + raw := flagVal + if raw == "" { + raw = os.Getenv(responseHeaderTimeoutEnv) + } + if raw == "" { + return defaultProxyResponseTimeout + } + d, err := time.ParseDuration(raw) + if err != nil || d <= 0 { + log.Printf("invalid response-header timeout %q, using default %s", raw, defaultProxyResponseTimeout) + return defaultProxyResponseTimeout + } + return d +} + // idleClientWriteTimeout bounds how long a single write of streamed response // bytes to the client may block. A killed client can leave a half-open socket // whose kernel send buffer fills and never drains; without this deadline the diff --git a/services/ollama-proxy/README.md b/services/ollama-proxy/README.md index 35f959b5..fcc9cfda 100644 --- a/services/ollama-proxy/README.md +++ b/services/ollama-proxy/README.md @@ -29,6 +29,7 @@ ollama-proxy [flags] | `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC | | `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. | | `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. | +| `--response-header-timeout` | *(`$NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT`, else `120s`)* | Upstream response-header timeout (Go duration, e.g. `5m`). A request whose engine has not sent response headers within this long fails with 502. Raise it for deep engine queues or slow model loads. | | `--version` | | Print version and exit | ### HTTP Reverse Proxy diff --git a/services/ollama-proxy/header_timeout_test.go b/services/ollama-proxy/header_timeout_test.go new file mode 100644 index 00000000..0b38bfcb --- /dev/null +++ b/services/ollama-proxy/header_timeout_test.go @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "testing" + "time" +) + +func TestResolveResponseHeaderTimeout(t *testing.T) { + cases := []struct { + name string + flag string + env string + want time.Duration + }{ + {"default", "", "", 120 * time.Second}, + {"env", "", "5m", 5 * time.Minute}, + {"flag beats env", "30s", "5m", 30 * time.Second}, + {"invalid env falls back", "", "bogus", 120 * time.Second}, + {"zero env falls back", "", "0", 120 * time.Second}, + {"negative flag falls back", "-1s", "", 120 * time.Second}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Setenv(responseHeaderTimeoutEnv, tc.env) + if got := resolveResponseHeaderTimeout(tc.flag); got != tc.want { + t.Fatalf("resolveResponseHeaderTimeout(%q) = %s, want %s", tc.flag, got, tc.want) + } + }) + } +} + +// The configured timeout must reach the upstream transports built after +// startup. +func TestProxyTransportUsesConfiguredHeaderTimeout(t *testing.T) { + old := proxyResponseTimeout + defer func() { proxyResponseTimeout = old }() + proxyResponseTimeout = 10 * time.Minute + tr := newProxyTransport(nil) + if tr.ResponseHeaderTimeout != 10*time.Minute { + t.Fatalf("ResponseHeaderTimeout = %s, want 10m", tr.ResponseHeaderTimeout) + } +} diff --git a/services/ollama-proxy/main.go b/services/ollama-proxy/main.go index afd147dd..29f90346 100644 --- a/services/ollama-proxy/main.go +++ b/services/ollama-proxy/main.go @@ -34,10 +34,13 @@ func main() { ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists") ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)") clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered") + responseHeaderTimeout := flag.String("response-header-timeout", "", "upstream response header timeout (Go duration, e.g. 5m); default: $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT or 120s") showVersion := flag.Bool("version", false, "print version and exit") resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo) flag.Parse() + proxyResponseTimeout = resolveResponseHeaderTimeout(*responseHeaderTimeout) + if *showVersion { fmt.Println(Version) os.Exit(0) diff --git a/services/ollama-proxy/proxy.go b/services/ollama-proxy/proxy.go index ad4ac7a2..fecb0509 100644 --- a/services/ollama-proxy/proxy.go +++ b/services/ollama-proxy/proxy.go @@ -19,6 +19,7 @@ import ( "net/http" "net/http/httputil" "net/url" + "os" "sort" "strconv" "strings" @@ -526,11 +527,16 @@ func (p *Proxy) Run(ctx context.Context) error { // Timeouts for upstream connections. Logged at startup so they're always // present in any captured log for post-mortem analysis. const ( - proxyDialTimeout = 10 * time.Second - proxyKeepAlive = 30 * time.Second - proxyResponseTimeout = 120 * time.Second - proxyMaxIdleConns = 50 - proxyIdleConnTimeout = 90 * time.Second + proxyDialTimeout = 10 * time.Second + proxyKeepAlive = 30 * time.Second + // defaultProxyResponseTimeout is the upstream response-header timeout + // unless --response-header-timeout (or NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT) + // overrides it. Ollama sends no headers until generation starts, so a + // request queued behind other work, or waiting on a slow model load, needs + // longer than this to survive the proxy. + defaultProxyResponseTimeout = 120 * time.Second + proxyMaxIdleConns = 50 + proxyIdleConnTimeout = 90 * time.Second // Inbound http.Server limits — keep IdleTimeout aligned with client // IdleConnTimeout so idle keep-alives are reaped on both sides. proxyReadHeaderTimeout = 10 * time.Second @@ -538,6 +544,38 @@ const ( maxModelListBytes = 16 << 20 ) +// responseHeaderTimeoutEnv carries the upstream response-header timeout when +// the --response-header-timeout flag is empty. The broker spawns the proxy as +// a child process, so the variable set on the broker (or desktop) is inherited +// without any broker change. +const responseHeaderTimeoutEnv = "NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT" + +// proxyResponseTimeout is the effective upstream response-header timeout, +// resolved at startup: --response-header-timeout flag, then +// NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT, then the 120 s default. Upstream +// transports are built lazily via newProxyTransport, so assigning it before +// serving is sufficient. +var proxyResponseTimeout = defaultProxyResponseTimeout + +// resolveResponseHeaderTimeout applies the flag > env > default precedence. A +// missing, unparseable, or non-positive value falls back to the default, so a +// bad setting can never silently disable the timeout. +func resolveResponseHeaderTimeout(flagVal string) time.Duration { + raw := flagVal + if raw == "" { + raw = os.Getenv(responseHeaderTimeoutEnv) + } + if raw == "" { + return defaultProxyResponseTimeout + } + d, err := time.ParseDuration(raw) + if err != nil || d <= 0 { + log.Printf("invalid response-header timeout %q, using default %s", raw, defaultProxyResponseTimeout) + return defaultProxyResponseTimeout + } + return d +} + // idleClientWriteTimeout bounds how long a single write of streamed response // bytes to the client may block. A killed client can leave a half-open socket // whose kernel send buffer fills and never drains; without this deadline the diff --git a/services/versions.json b/services/versions.json index 29d8c230..d6840157 100644 --- a/services/versions.json +++ b/services/versions.json @@ -3,8 +3,8 @@ "product": "0.91.7", "installer": "0.91.7", "components": { - "ollama-proxy": "0.26.2", - "lmstudio-proxy": "0.16.2", + "ollama-proxy": "0.27.0", + "lmstudio-proxy": "0.17.0", "nvpair-node-info": "0.13.3", "nvpair-node-scanner": "0.20.3", "nvpair-manual-nodes": "0.11.1",