fix(client): overlapping processes using one saved key stall an earlier SSH session #162

Summary

An SSH session through Tailcat can stop making progress after a second Tailcat client process using the same saved client identity connects to the same server and exits. A plain-SSH test reproduces a temporary stall with the release and current default-branch builds; it needs neither scp, a PTY, SSH multiplexing nor a server restart. The controls without the second process complete promptly.

Steps to reproduce

Use a throwaway SSH key and a throwaway saved Tailcat client identity, with an SSH server provided by tailcat serve ssh. Allowlist that client's node public key and authorize the throwaway SSH public key. Keep the server running throughout each trial. The local tests used a Linux container on the client's Mac; a separate-network observation is described below.

1. On the client, create a scratch directory with work=$(mktemp -d), generate an SSH key with ssh-keygen -t ed25519 -N '' -f "$work/ssh-test", and generate the client identity with tailcat genkey --client --key="$work/client.private.json". Transfer only the two public keys to the test server.
2. On the server, run tailcat serve --key=new --allow="$CLIENT_NODE_PUBLIC_KEY" --ssh-authorized-keys="$AUTHORIZED_KEYS_FILE" ssh, where the two variables contain the public key printed by client generation and the file containing the test SSH public key. Keep the printed Tailcat address private and set it as TC in the client shell.
3. Create this client configuration, expanding the scratch directory and address locally:

cat > "$work/ssh.conf" <<EOF_CONFIG
Host test-server
  HostName test-server
  User test
  IdentityFile $work/ssh-test
  IdentitiesOnly yes
  IdentityAgent none
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  GlobalKnownHostsFile /dev/null
  BatchMode yes
  ConnectTimeout 20
  ControlMaster no
  ProxyCommand tailcat --key=$work/client.private.json $TC 22
EOF_CONFIG

4. Create the FIFO once, then use the A-start commands below for an initial no-B control. Confirm A-up before closing its stdin as in step 5, with the B command omitted, and confirm A exits zero. Then rerun the A-start commands using the existing FIFO and the same saved client key, again waiting for A-up, to prepare the overlap trial:

mkfifo "$work/input"
ssh -vvv -F "$work/ssh.conf" test-server \
  'echo A-up; cat >/dev/null; echo A-done' \
  <"$work/input" >"$work/A.out" 2>"$work/A.log" &
a=$!
exec 9>"$work/input"
# Wait for A-up in A.out before starting B.

5. Run B to completion, wait three seconds, then close A's stdin:

ssh -n -F "$work/ssh.conf" test-server true
sleep 3
exec 9>&-
wait "$a"

6. Use one fresh client identity per control/overlap pair: the completed no-B control, overlap A, and overlap B all use that same identity, with no server restart between them. Repeat for three client identities. Repeat the set with TS_DEBUG_ALWAYS_USE_DERP=1 in both server and client environments to disable peer UDP traffic.

The executed matrix generated three fresh saved client identities per build/transport setting, ran all three no-B controls first, then reused the respective identities for all three overlap trials and all three scp trials while keeping that setting's server running. Within each overlap trial, A and B shared the same saved identity. The harness used Python pipes rather than a FIFO to hold A's stdin open, waited for A-up, waited for B's observed exit, then closed A's stdin after three seconds. The shell recipe expresses the same per-identity ordering; this investigator did not rerun its literal FIFO syntax. The harness also used a saved throwaway server key rather than an ephemeral server key.

Expected behavior

After A's stdin closes, the server command should receive EOF, print A-done, and return an SSH exit status and channel close. Running another authorized command with the saved client identity should preserve that progress. The documented saved-identity workflow automatically reuses a persistent client key for client commands; no one-process limitation is documented. If overlapping use is intentionally unsupported, an explicit restriction or observable rejection would clarify that contract instead of leaving an established session silently stalled.

Actual behavior

Each matrix cell ran three no-B controls, three same-key overlap trials, and three two-source scp trials:

Two additional forced-DERP trials, each with its own passing no-B control and a 65-second timeout, recovered without intervention: the release at 12.881 s and the default branch at 15.188 s after stdin EOF. In one further default-branch forced-DERP contrast trial, B used a different allowlisted client identity and A completed in 0.071 s; its no-B control took 0.062 s. Every B exited zero.

An additional sequential-reuse control set used three identities per build/transport setting and ran two no-B sessions with each key against an unchanged server. All 24 sessions exited zero after stdin EOF; all 12 second-use sessions also completed promptly, in 0.002–0.080 s. This checks reuse after a completed session separately from overlapping processes.

In every short-window stalled trial, A printed only A-up. B exited successfully. After stdin closed, A's verbose SSH trace reached the following point and received neither an exit status nor a channel close before the observation cutoff:

debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed

The controls and the recovered long-window trials continued with:

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug1: Exit status 0

The local tests establish a temporary progress failure, not an indefinite local hang. The long-window trials were given a 65-second timeout but completed earlier; no claim is made that a locally tested A remained stuck for 60 seconds.

Evidence

• Expected source: The README's Protected SSH server over DNS example documents a saved client identity, allowlisting, and automatic reuse by client modes. In the tested source, cmd/tailcat/tailcat.go:795-817 loads that saved key for each invocation. This supports the saved-key input; the demonstrated control supplies the expected SSH EOF/exit behavior. Concurrent-process support is implicit in that workflow rather than an explicit documented guarantee.
• Failure source: This investigator's plain-SSH trials and inline trace excerpts above; A is already executing before B starts, and the server stays alive throughout. The separate cross-network scp observation below comes from supplied saved process snapshots and SSH traces.
• Evidence provenance: observed
• Local verification: reproduced
• Reproduction completeness: complete

These fields describe the directly observed temporary progress stall, with successful no-B controls. The separately attributed cross-network scp observation is supplied evidence; its leftover was not reproduced locally and its persistence and shared cause remain unverified here.

Restoration check

Repeat the same ordered two-session test and its no-B control with an unchanged server. In both normal and forced-DERP operation, A should receive its command output, exit status and channel close promptly after stdin EOF despite B having connected and exited. Verify the real stream and both SSH exits; a test that merely terminates A or omits the overlap would mask the failure. Check the two-source scp workflow separately for leftover SSH and Tailcat processes, since copying the files and returning zero alone does not establish cleanup.

Runtime or environment

• Tested release: Tailcat v0.7.0 on both sides, the latest published release at verification. The Mac client was an isolated copy of the Homebrew binary; the Linux server was the official arm64 release archive, verified against its published SHA-256 manifest.
• Tested default branch: main at e00cdde6ff787f801361ff85ed085775c775d4c5, clean source builds for macOS arm64 and Linux arm64, built with Go 1.27.3. Both roles used the matching build in each trial.
• Local client: macOS 27.2, build 26B123, arm64; OpenSSH_10.5p1, LibreSSL 3.3.6.
• Local server: Debian 13.7 arm64 in Docker Desktop, Docker Engine 29.9.3. Ordinary mode permitted direct UDP but its selected path was not measured. Forced-DERP mode set the documented dependency debug knob in both processes; no host network settings were changed.
• Supplied cross-network server: Debian 13 Linux amd64 with the official release binary. It was on another network and reached through a DERP relay; whether a direct path subsequently formed was not measured. Its client used the same release and OpenSSH/platform baseline stated above.

Additional context

A supplied separate-network run of scp -F conf test-server:/dir/a.txt test-server:/dir/b.txt out/ copied both files and returned zero in six seconds, but one SSH process and one Tailcat client remained. Saved process snapshots show the SSH process reparented to PID 1 and still alive 18 seconds after the copy began, until test cleanup stopped its process group. Its trace ended after send eof; the second connection received exit-status, channel close and exit status zero. Single-source scp and a standalone ssh -s test-server sftp </dev/null control exited normally and left no clients in the supplied run. In all 12 local two-source scp trials, both fixture files matched byte-for-byte, scp exited zero, and the process group had no SSH or Tailcat processes left three seconds later. The common trigger makes this useful context, but the local tests do not establish that the leftover process and the temporary plain-SSH stall share an implementation cause. No cause or fix is asserted.

References

• Issue #159 concerns a client that stops working after a server restart; this report keeps the server running and starts the failure with a second client process.
• Issue #151 concerns netstack close after hundreds of TCP endpoints. Its dependency fix is already in the tested default branch through merged PR go.mod: include Tailscale netstack close fix #161; this test needs only two overlapping SSH sessions.
• Merged PR #83 concerns proxied TCP teardown and is already included in both tested builds. A shared cause has not been established.

Triage assessment

• Impact level: P2
• Assessment status: supported
• Impact basis: Overlapping authorized client processes using one saved identity can silently stall an existing SSH command, even after the second command exits. Local trials demonstrate delays well above the passing controls; the supplied scp run additionally shows successful copy exit without prompt subprocess cleanup. Local indefinite failure and population frequency are not established.
• Workaround status: available
• Workaround basis: Serialize client processes sharing an identity; all no-B local controls complete promptly. This preserves command output and completion, although it prevents overlap. One different-key contrast trial also passed, but that single result does not establish a general workaround guarantee.