Skip to content

perf: Disable Nagle (TCP_NODELAY) on SSH transport to reduce per-command latency - #52

Open
ssalin2 wants to merge 1 commit into
intel:mainfrom
ssalin2:perf/ssh-tcp-nodelay
Open

ssalin2 wants to merge 1 commit into
intel:mainfrom
ssalin2:perf/ssh-tcp-nodelay

Conversation

@ssalin2

@ssalin2 ssalin2 commented Oct 8, 2026

Copy link
Copy Markdown

What

Set TCP_NODELAY=1 on the SSH transport socket in SSHConnection._connect.

Why

paramiko leaves TCP_NODELAY unset, so the small exec-request and exit-status
packets each hit a TCP delayed-ACK stall (~40 ms) on every execute_command.
Disabling Nagle removes the stalls.

Measured (loopback SSH, network RTT ~0, so this is pure protocol overhead)

per-command median
before (TCP_NODELAY=0) 89.9 ms
after (TCP_NODELAY=1) 49.6 ms (−44%)

Phase breakdown before the fix: exec_command ~43 ms + recv_exit_status ~45 ms
(the two Nagle/delayed-ACK round trips); open_session was only ~1.3 ms — i.e. the
cost was the stalls, not channel creation. On suites that issue thousands of commands
per test this is minutes of wall-clock savings.

Scope / safety

  • Affects packet batching only — command execution, timeout handling, return codes
    and captured stdout/stderr are unchanged.
  • Placed in _connect so it also applies after _reconnect().
  • Guarded (transport is not None and transport.sock is not None) so it can't raise.
  • import socket is stdlib; no new dependency.

Draft for review / to run CI. Happy to add a unit-test assertion that
setsockopt(IPPROTO_TCP, TCP_NODELAY, 1) is invoked on connect if desired.

… latency

paramiko leaves TCP_NODELAY unset on the transport socket, so the small
exec-request and exit-status packets each hit a TCP delayed-ACK stall
(~40 ms) on every execute_command. Setting TCP_NODELAY=1 right after
connect() (and on reconnect, since _connect runs there too) removes the
stalls.

Measured on a loopback SSH connection (network RTT ~0):
  per-command median  89.9 ms -> 49.6 ms  (-44%)
Breakdown showed ~43 ms in exec_command and ~45 ms in recv_exit_status,
both eliminated by disabling Nagle.

This only affects packet batching; command execution, timeout handling,
return codes and captured output are unchanged.

Signed-off-by: Sam Salin <samuel.salin@intel.com>
@coveralls-official

Copy link
Copy Markdown

Coverage Status

coverage: 80.839% (+0.01%) from 80.829% — ssalin2:perf/ssh-tcp-nodelay into intel:main

@ssalin2
ssalin2 marked this pull request as ready for review October 27, 2026 14:16
Copilot AI balanced review requested due to automatic review settings October 27, 2026 14:16
Comment thread mfd_connect/ssh.py
# exec/exit-status packets hit TCP delayed-ACK stalls (~40 ms each), roughly doubling
# per-command latency. This changes packet batching only - command semantics,
# timeouts, return codes and output are unaffected.
transport = self._connection.get_transport()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one doubt. In case of using SSHConfigConnection with ProxyJump, transport.sock is a Paramiko Channel, not a native TCP socket. SSHConfigConnection._connect() passes the proxy channel via the sock argument, and Paramiko retains it as transport.sock.

Since Channel does not implement setsockopt(), it passes the None checks but raises AttributeError: 'Channel' object has no attribute 'setsockopt' here after successful authentication. This breaks initialization and reconnection for ProxyJump connections using the normal authentication path.

Could we restrict this optimization to native TCP sockets and add a regression test with a channel-backed transport? The existing test_connect_injects_sock_for_proxied mocks SSHConnection._connect(), so it does not exercise this new call.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Disables Nagle’s algorithm (sets TCP_NODELAY=1) on the Paramiko transport socket to reduce per-command SSH latency caused by delayed-ACK stalls.

Changes:

  • Import stdlib socket to access TCP_NODELAY.
  • After connecting, fetch Paramiko transport and set TCP_NODELAY on its underlying socket when available.
File Description
mfd_connect/​ssh.py Sets TCP_NODELAY on the SSH transport socket right after connect to improve command latency.

Review effort: Lite

Comment thread mfd_connect/ssh.py
# timeouts, return codes and output are unaffected.
transport = self._connection.get_transport()
if transport is not None and transport.sock is not None:
transport.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
Comment thread mfd_connect/ssh.py
Comment on lines +142 to +146
# per-command latency. This changes packet batching only - command semantics,
# timeouts, return codes and output are unaffected.
transport = self._connection.get_transport()
if transport is not None and transport.sock is not None:
transport.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants