[Bug]: ollama-proxy: undocumented 120 s upstream response-header timeout turns a queued or cold-loading request into a 502

[msb-msb]

Summary

ollama-proxy returns 502 {"error":"upstream error: net/http: timeout awaiting response headers"} on any upstream request that has not produced response headers within 120 s. Ollama serves one request at a time by default (OLLAMA_NUM_PARALLEL=1) and sends no headers until generation starts, so a request that sits in Ollama's queue for more than 120 s, or that waits on a model load longer than 120 s, fails through PAIR while the identical request sent directly to Ollama succeeds. The limit is not mentioned in the README, services/ollama-proxy/README.md, the flag table, or docs/troubleshooting.mdx, and there is no flag or setting that changes it. 120 s is the observed value; I did not locate the constant in the source.

PAIR version or commit

v0.1.1 release archive service-binaries-linux-x64.zip: product 0.91.7, ollama-proxy 0.26.2, nvpair-ui-broker 0.40.2, run headless via nvpair-tui 0.7.2 in tmux with NVPAIR_LOG_LEVEL=debug.

Affected component

ollama-proxy

Environment

• Node A: Ubuntu 24.04.4, kernel 6.8.0-138, RTX 3090 24 GB, Ollama 0.30.0
• Node B: Ubuntu 24.04.4, RTX 3060 12 GB, Ollama 0.17.5
• Two nodes paired. Ollama was pre-existing on both and adopted by PAIR on 11434; proxy on 11435.

Steps to reproduce

1. Queue depth (single node). Ollama on node A with qwen3.6:27b Q4_K_M (about 11 s per 200-token request on the 3090). Fire 20 independent POST /v1/chat/completions concurrently at the PAIR proxy (127.0.0.1:11435), max_tokens: 200, stream: false. Fire the same 20 at Ollama directly (127.0.0.1:11434) as the control.

Path	Wall clock	Succeeded	Failed
Direct to Ollama	226.5 s	20	0
Through PAIR	120.3 s	10	10, all 502 at 120.24 to 120.31 s

Per-request completion times through PAIR: 12.7, 23.7, 35.2, 48.4, 58.5, 70.7, 81.5, 92.8, 104.5, 115.6 s, then ten failures at 120.2 to 120.3 s. Every request whose generation would have started after 120 s was dropped. GPU utilization on the node fell to zero within one second of the 502s: Ollama cancelled the orphaned queue entries when the proxy closed the connections.

2. Same test, 8 concurrent instead of 20, so the deepest queue position starts before 120 s:

Path	Wall clock	Succeeded	Failed
Direct to Ollama	86.5 s	8	0
Through PAIR	85.6 s	8	0

Per-request completions through PAIR: 11.4, 22.4, 32.6, 42.6, 52.9, 64.4, 75.1, 85.6 s. No 502s. The only variable between this run and the 10-of-20 failure is queue depth.

3. Cold load (cluster, model only on the peer). Node B holds deepseek-r1:14b (9 GB) on a slow disk; cold load takes 2 to 3 minutes. From node A, POST /v1/chat/completions for that model to node A's proxy. PAIR routes it to node B correctly, then 502s at 120 s. Node B's Ollama abandoned the load when the connection dropped (/api/ps empty afterwards). A retry 30 s later succeeded in 31.6 s because the file was by then in page cache.

Expected behavior

Either no fixed header timeout on inference routes, or a default at least as long as Ollama's own OLLAMA_LOAD_TIMEOUT (5 minutes) with a way to configure it (flag on ollama-proxy, settings.json, or a per-engine setting), plus a line in the proxy README and troubleshooting doc stating the default and the error text it produces.

Actual behavior

A silent 120 s limit that fails requests the engine would have served. Because the scheduler forwards immediately rather than queueing, the effective maximum burst size through PAIR is 120 s / per-request time per node, regardless of how many requests the client sends. For the 27B example that is about 10 requests per node. Workaround: keep bursts under that ceiling, or pre-load models with a direct keep_alive request before routing through PAIR.

Sanitized logs

One line per failed request, queue-depth case:

12:04:58.xxx [ollama-proxy] DEBUG proxy request complete id=... node_id=a9489f3b-... method=POST path=/v1/chat/completions target=127.0.0.1:11434 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Cold-load case:

11:00:20  request sent
11:02:20.115 [ollama-proxy] DEBUG proxy request complete id=65 node_id=70506f7d-... method=POST path=/v1/chat/completions target=192.168.1.32:11435 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Full request logs, the load generator, and GPU samples are available on request.

[kvnloo]

+1 on raising or configuring the 120 s ResponseHeaderTimeout. Your queue-depth and cold-load reproduction is clean — I hit the identical wall adopting existing Ollama on :11434 in a headless two-node cluster: any model load longer than 120 s upstream → 502 through PAIR, success direct.

The 120 s constant (proxyResponseTimeout in services/ollama-proxy/proxy.go) predates any flag for it, and the README + troubleshooting docs are silent on the ceiling. For adopt-existing Ollama the workaround is pre-loading models with a direct keep_alive request before routing through PAIR, but that defeats zero-config adoption.

Headless deployments especially need docs on non-default proxy bind / port-collision handling when the broker can't claim :11434 because Ollama already holds it — the fallback is :11435 but that's not written down anywhere outside the source.

Soft +1 on #52 (configurable --response-timeout flag). No competing PR — it's a clean, minimal fix that defaults to current behavior and would unblock both the queue and the cold-load cases.

[Noah-Tervalon-Nvidia]

We do want to include this, we're looking into allowing configuring the header and also allowing changes to launch options on the engines.