Cancel non-streaming chat inference when clients disconnect

[Baiju Meswani (baijumeswani)]

Summary

• Monitor the non-streaming Chat Completions TCP connection during synchronous inference without reading request bytes, and cancel the active request on peer disconnect.
• Release request-scoped session and Engine state after cancellation and classify disconnected route telemetry as canceled rather than a successful completion or inference failure. Streaming behavior and other endpoints are unchanged.
• Cover buffered-input/keep-alive safety, stop-on-completion, an actual HTTP disconnect with canceled telemetry and a successful follow-up request.

Verification

• Built via python sdk_v2/cpp/build.py on Linux.
• Focused monitor, HTTP disconnect, and pre-existing streaming-disconnect tests passed.

Normal TCP peer disconnects are in scope; silent network blackholes without FIN/RST require a separately specified request deadline. Windows socket handling is included but was not exercised on a Windows runner.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Oct 13, 2026 7:06am UTC

[Copilot]

🟡 Changes recommended

Windows disconnect detection fails when unread input precedes the FIN, and the telemetry assertion is race-prone.

2 open findings

• Detect Windows FIN despite buffered unread data · New
• Wait for canceled action before asserting telemetry snapshot · New

What changed in this PR

Adds client-disconnect cancellation for synchronous Chat Completions inference.

Changes:

• Adds a TCP disconnect monitor that cancels active requests.
• Reports disconnected requests as canceled telemetry.
• Adds monitor and end-to-end disconnect tests.

File	Description
sdk_v2/​cpp/​src/​service/​non_streaming_disconnect_monitor.h	Implements cross-platform peer monitoring.
sdk_v2/​cpp/​src/​service/​chat_completions_handler.h	Extends non-streaming handler inputs.
sdk_v2/​cpp/​src/​service/​chat_completions_handler.cc	Integrates cancellation and telemetry classification.
sdk_v2/​cpp/​test/​internal_api/​web_service_test.cc	Adds disconnect, recovery, and telemetry tests.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Monitor shutdown can miss a disconnect occurring after its final poll, incorrectly recording success.

0 open findings

2 resolved since last review

• Wait for canceled action before asserting telemetry snapshot
• Detect Windows FIN despite buffered unread data

Previously missed (1)

In code that hasn't changed since last review

Stop can miss disconnect before reporting successful route

sdk_v2/​cpp/​src/​service/​non_streaming_disconnect_monitor.h:64

Stop() sets the stop token before guaranteeing one final socket check. A FIN/RST can therefore arrive after the worker's last poll (or before the new thread first runs), then Stop() wakes/joins the worker and returns false; the handler records a successful route even though the peer disconnected during inference. Please synchronize shutdown with a final nonblocking PeerDisconnected check (while retaining the connection/request safely), and add a regression test for disconnect-immediately-before-stop.

🧠 Review effort: Balanced

[Baiju Meswani (baijumeswani)]

Addressed the new Copilot review finding in 2b47789. Stop() now joins the worker and performs one final nonblocking peer check while the monitor still owns the TCP connection and request; repeated Stop calls do not recheck a completed request. Added an immediate-disconnect-before-Stop regression (and made the Windows buffered-input case exercise immediate Stop). Focused disconnect and existing streaming tests passed in 10 repeated runs.

[Copilot]

🔵 Needs a closer look

Low-level cross-platform socket and concurrency behavior warrants human review, particularly because the Windows path was not executed.

0 open findings

🧠 Review effort: Balanced