A frozen node is twice as expensive as an absent one, and marks its peer's completed work as failed

[canja006]

Two observations from the same failure mode, measured on a two-node cluster (Ubuntu + RTX 3060, macOS, v0.1.1-463). Both matter for a cluster that includes a laptop, since a laptop that sleeps mid-request is exactly this shape: the socket is accepted, nothing answers.

Setup

24 concurrent chat/completions, max_tokens: 200, google/gemma-4-12b resident on both nodes.

Condition, same load	Wall clock
Both nodes healthy	37.8 s
Single node, bypassing the router entirely	72.8 s
Peer frozen (SIGSTOP on its nvpair processes)	155.6 s

Every request returned 200 in all three. Reliability is not the problem — the cost is.

A frozen peer is 2.1× worse than not having it at all, and 4.1× worse than a healthy cluster. When I instead killed the peer's processes outright, the same load finished in 28.3 s with no failures: a closed socket gives an immediate RST and the proxy moves on in milliseconds. It is specifically the accepted-but-silent socket that hurts.

For context on the eviction path: after the peer was killed, discovery had dropped it from the node list within 75 s — I sampled too coarsely to pin it more precisely. During that window each request pays its own connection timeout.

The second half: false failures

When the frozen peer resumed, its job history marked several of the other node's workloads as

Failed — node serveri stopped reporting this workload

Those workloads had all completed successfully; the requesting node returned 200 for all 24. The peer's view had simply gone stale while it was stopped, and on resume it reaped them as failed. The timestamps match the resume to the second.

That leaves the job history — the only place a user sees what the cluster did — showing failures that did not happen, precisely in the situation where something unusual was going on and they would be looking at it.

Not filed as a defect

Neither is a crash and both may be deliberate. But if a laptop is a first-class node, asleep with the socket accepted and nothing answering is its most likely state, not an edge case. Two directions that would help, if either is welcome:

• a connection-level timeout distinctly shorter than the request timeout, so an unresponsive candidate is abandoned quickly while a slow-but-live generation is not
• reaping a peer's workloads as unknown rather than failed when the gap was on our side

Happy to open a PR for either if you have a preferred shape.

[Noah-Tervalon-Nvidia]

We need to look into this a bit more. We intentionally let frozen nodes not kill connections and it's hard to distinguish between a node that is either too busy to respond, frozen, or actually gone. Open to suggestions and can elaborate a bit more on why we made the decisions we did, but the current state is reflective of our first pass at this problem. There is definitely more work that can be done on this front.

[canja006]

Thanks — and I'll start by withdrawing half of my own suggestion, because reading the code shows it wouldn't have worked.

A connect-level probe cannot see a frozen node. reach.accepts completes a TCP handshake and closes ("the handshake is the entire question"), which is the right question for which address of a multi-homed peer to use. But a SIGSTOPed process still has its listening socket in the kernel, and the kernel completes handshakes without userspace ever calling accept. A sleeping laptop is the same shape. So a shorter connect timeout would have returned "reachable" just as fast, and my first bullet was wrong. Your point about the three states being hard to tell apart is exactly right at that layer.

What does separate them is time to first response header. That boundary is already in the code — proxyResponseTimeout (120 s) is set as ResponseHeaderTimeout on the forwarding transport, and ModifyResponse is described as "both the retry decision point and, on commit, the time-to-first-byte boundary". The reason it discriminates where a connect probe can't: a slow-but-live generation is slow in the body. LM Studio sends the status line and headers before the first token, so a busy node still answers the header promptly while a frozen one sends nothing at all. That makes the header deadline a lever you can set far shorter than a whole-request budget without ever punishing a slow generation — which is the specific thing I understand you didn't want to break. I'm not going to propose a number; you know the congested-VPN cases better than my LAN does.

But the 2.1× isn't really the timeout — it's that nothing remembers. As far as I can see there's no failure memory anywhere in candidate ordering: resolveCandidates orders by the scheduler's priority list and then stable ID, and I find no cooldown, demotion or backoff for a candidate that just timed out. So my 24 concurrent requests each independently chose the frozen peer first and each paid the full deadline. One observed first-byte timeout demoting that candidate for a short window would turn N × timeout into 1 × timeout, and it decides nothing about whether the node is gone — it only says "the last attempt against this one cost us everything, put it second for a while". That alone would take the frozen-peer case below the single-node baseline, which is the part that felt worst: the cluster being slower than no cluster.

On the false failures, the machinery is already right — it's the label that reaches the user. failStaleForeignWorkloads applies its verdict as inferred, guarded on the sighting it came from, so the origin's authoritative event wins on return; the comments make the reasoning explicit. What the user sees, though, is Failed — node X stopped reporting this workload, indistinguishable from a real failure, in exactly the moment they'd be looking at the history because something odd happened.

Two cheap discriminators for "the gap was on our side", either of which would have suppressed my case, since it was our process that was stopped:

• The sweep's own tick arriving late. A stopped or slept process resumes with a tick that is minutes overdue against the wall clock. That is a direct observation of our own outage, not an inference about theirs — skip the sweep and restart the clock.
• Nobody spoke, not just this origin. If no event arrived from any foreign origin across the window, every peer went silent simultaneously, which is us.

And if the verdict is still wanted after that, a label distinct from failed — "lost contact", or a non-terminal unknown — keeps the history honest without losing the signal.

Happy to prototype the cooldown as a PR, since it's the smallest and most contained of these and doesn't commit you to a liveness model. Equally happy to hear the elaboration you offered first — if the intended direction is different, I'd rather write against it than around it. And I can re-run the measurements on the two-node setup for anything you'd like numbers on.

[mkalkere]

Read the code on this. I think there are two separate mechanisms:

1. The 2.1x cost: candidate selection has no failure memory. A frozen peer's kernel still completes TCP handshakes, so the reachability probe (which only answers the which-address question) reports reachable, and then every request independently eats the full 120s header deadline before failing over. A cooldown on candidates that just burned a full deadline would cut the repeated cost without changing the intentional "don't kill connections" behavior.

2. The false failed labels: those come from the broker's stale-workload sweep running on the frozen node itself after it wakes up. After 5 minutes of silence the first tick condemns workloads the peer actually finished. An own-outage guard -- a node coming back from silence shouldn't condemn others' work on its first sweep -- looks strictly safe.

Neither requires distinguishing busy/frozen/gone, which was the core concern. canja006 already offered to prototype the cooldown; happy to take the sweep guard if you want it split up. What's the intended direction?

[canja006]

Thanks for reading the code rather than taking my word for it, and the split works for me. I'll take the cooldown. The sweep guard is yours as far as I'm concerned, and your framing of it is better than mine: a node coming back from its own silence is direct evidence about itself, and needs no inference about anybody else.

One thing I found after my last comment strengthens the case for the cooldown, and corrects something I got wrong in it.

I claimed the time-to-first-header deadline is a clean discriminator, on the grounds that a slow-but-live generation is slow in the body while a frozen node sends nothing at all. That holds for lmstudio-proxy. It does not hold for ollama-proxy: #57 is precisely a request that legitimately produces no headers for more than 120 s, because Ollama sends none until generation starts and serves one request at a time by default. So the same deadline is at once too long for my frozen peer and too short for that user's cold load, and #83 makes it configurable without answering which of the two an operator is supposed to tune for. A single number cannot serve both.

That is the argument for the cooldown. It needs no number and no liveness model. It says only that the last attempt against this candidate cost a full deadline, so try it second for a while.

The shape I had in mind, so there is something concrete to say no to:

• trigger on a first-byte timeout specifically, not on any error. A 500 from a live node is not this.
• demote, never exclude. A demoted candidate stays in the list and is still tried when it is the only one, so a single-node cluster is unaffected.
• per-process and in-memory, one small fixed window, no state shared between proxies and nothing persisted.
• expire on time alone, and let the next success clear it, so a peer that comes back is not held down.

@Noah-Tervalon-Nvidia you offered to elaborate on why the current behaviour is what it is. I would rather hear that before sending a PR that argues past it. Is the shape above in the direction you would want, or is there a reason the ordering should stay memoryless that I am not seeing?