proxies: raise the upstream response-header timeout to 30m and make it configurable

[cguldogan]

Description

ollama-proxy and lmstudio-proxy forward inference requests over an http.Transport with ResponseHeaderTimeout: 120s. That timer runs from the moment the request is sent until the first response header byte. A streaming completion sends headers immediately, so it never notices. A non-streaming completion sends nothing until the whole answer is generated, so the timer covers queue wait + prefill + the entire generation — and any non-streaming request that queues behind other jobs, or runs a reasoning model, fails after exactly 120 s:

502 {"error":"upstream error: net/http: timeout awaiting response headers"}

The job is marked failed and the engine's work up to that point is discarded. A request to a PAIR peer crosses two hops (router → peer's mTLS ingress → engine), and each hop applied the same 120 s independently.

Every SDK call without stream: true is non-streaming — the OpenAI SDK's default chat.completions.create(...), most eval and batch scripts. Their own default timeouts are 600 s (OpenAI SDK, LiteLLM); PAIR was cutting off at 120.

Change: the default becomes 30 minutes, and it is configurable per install:

• --response-header-timeout <duration> on both proxies, or
• NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT in the environment (the proxies inherit the broker's environment, so this reaches them on desktop and headless installs without a spawn change),

with the same precedence as --log-level (flag > env > default). 0 waits indefinitely. Unreachable hosts are still cut off by the 10 s dial timeout, dead clients by idleClientWriteTimeout, and a client that gives up cancels the upstream request through its context, so the header deadline only remains as a backstop for an engine that accepted the connection and never answers. Model-list fetches keep their separate 10 s limit.

Scope

• services/shared/envflag: new small package — a duration flag whose default comes from an NVPAIR_* env var, mirroring applog.RegisterFlag.
• Both proxies: defaultResponseHeaderTimeout (30 m), a responseHeaderTimeout field on Proxy threaded into newProxyTransport (plain, manual and mTLS peer transports alike), the flag/env wiring in main.go, and the startup "proxy timeouts configured" log line reporting the effective value.
• READMEs (flag tables) and services/readme.md (new "Inference timeouts" section).
• services/versions.json: MINOR bump for ollama-proxy (0.26.2 → 0.27.0) and lmstudio-proxy (0.16.2 → 0.17.0) — each gains a new flag and a changed default a --version reader would want to know about; product/installer 0.91.7 → 0.92.0 per VERSIONING.md (at least one MINOR component bump, user-visible behaviour change).
• Not in scope: no JSON-RPC or settings-UI surface (that would pull in the broker relay, the service bridge and the generated contract docs); the env var covers the operator case.

Validation

• cd services/shared && go test ./envflag/ — precedence (flag > env > default), 0, and error messages that name their origin (--flag vs $ENV).
• cd services/lmstudio-proxy && go test ./... and the same in ollama-proxy — new response_header_timeout_test.go in each:
  • the configured value reaches the plain and peer transports, and 0 disables the deadline;
  • a silent httptest upstream (500 ms before headers) yields 502 with a shorter deadline and 200 with a longer one or 0 — i.e. the deadline is really applied end to end through handleHTTP.
• go test -race on both proxies; go vet on all three modules; make headers, make contracts clean.
• cd services/tests && go test ./... (cross-process suite): pass, no skips, with the desktop app stopped and an isolated HOME.
• Manual: --help shows the flag with (default: $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT or 30m0s); NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT=bogus exits with $NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT: "bogus" is not a duration (use e.g. "30m", "90s" or "0").
• Pre-existing, unrelated: ollama-proxy's TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone fails on a stock macOS host because it binds 127.0.0.2, which is not aliased on loopback by default; it fails identically on an untouched checkout of the base.

Verification ceiling: the first hop is unit-tested as above. That the peer's ingress hop applies the same deadline is read from ingress.go (newLocalReverseProxy → plainHTTPTransport()), not measured end to end across two machines.

Risk

• Behaviour change: a non-streaming request against a hung engine now waits up to 30 min (or the client's own timeout, whichever comes first) instead of 120 s before the proxy reports 502. With the proxy no longer the binding limit, the caller's timeout becomes it — a client with a 120 s SDK timeout still fails, just not because of PAIR.
• Both hops must be upgraded for the two-hop case to benefit; a peer still on the old binary keeps its 120 s.
• No protocol, storage or packaging changes. New env var name: NVPAIR_PROXY_RESPONSE_HEADER_TIMEOUT.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

🤖 Generated with Claude Code