[Bug]: SSHTunnelPool races mkdir against rmtree on a shared connection dir, flaking tests under -n auto

[peterschmidt85]

Steps to reproduce

SSHTunnelPool creates and removes the same connection directory from different processes, and mkdir(exist_ok=True) is not atomic against a concurrent rmtree. Under pytest -n auto this fails test_terminating_jobs.py::TestJobTerminatingWorker::test_terminates_job[sqlite].

The race is between two lines that operate on the same conn_dir:

• pool.py:317 conn_dir.mkdir(parents=True, exist_ok=True) in _resolve_conn_dir
• pool.py:282 shutil.rmtree(self._real_conn_dir, ignore_errors=True) in close

exist_ok=True does not make mkdir atomic. CPython swallows FileExistsError only if the path is still a directory when it re-checks:

# pathlib.py
try:
    self._accessor.mkdir(self, mode)
except OSError:
    if not exist_ok or not self.is_dir():
        raise

So mkdir raises because the directory exists, close() removes it before is_dir() runs, and the error propagates.

To reproduce the race directly, run this against any directory:

import shutil, threading
from pathlib import Path

target = Path("/tmp/conns/127.0.0.4:22,10999,10998")
target.parent.mkdir(parents=True, exist_ok=True)
stop, errors = threading.Event(), []

def maker():
    while not stop.is_set():
        try:
            target.mkdir(parents=True, exist_ok=True)
        except FileExistsError as e:
            errors.append(e); stop.set(); return

def remover():
    while not stop.is_set():
        shutil.rmtree(target, ignore_errors=True)

ts = [threading.Thread(target=maker) for _ in range(4)]
ts += [threading.Thread(target=remover) for _ in range(4)]
for t in ts: t.start()
stop.wait(20); stop.set()
for t in ts: t.join()
print(errors[0] if errors else "no error")

It fails within a second, every run.

Three conditions make the test suite hit it:

1. CONNECTIONS_DIR is SERVER_DIR_PATH / "instance-connections", the real ~/.dstack/server/, shared by every process on the machine rather than a tmp dir.
2. The directory name is {hostname}:{port},{forwarded_ports}, and tests use fixed values, so workers collide on the identical path.
3. CI runs uv run pytest -n auto src/tests --runui, so those workers are concurrent processes.

Actual behaviour

FAILED src/tests/_internal/server/background/pipeline_tasks/test_terminating_jobs.py::TestJobTerminatingWorker::test_terminates_job[sqlite]
  - FileExistsError: [Errno 17] File exists: '/Users/runner/.dstack/server/instance-connections/127.0.0.4:22,10999,10998'

src/dstack/_internal/server/background/pipeline_tasks/jobs_terminating.py:692: in _process_terminating_job
src/dstack/_internal/server/background/pipeline_tasks/jobs_terminating.py:885: in _stop_container
src/dstack/_internal/server/services/runner/ssh.py:89: in wrapper
src/dstack/_internal/server/services/runner/pool.py:113: in get_or_open
src/dstack/_internal/server/services/runner/pool.py:201: in __init__
src/dstack/_internal/server/services/runner/pool.py:317: in _resolve_conn_dir
pathlib.py:1175: FileExistsError

It is scheduling dependent, so it appears on one matrix job while the same commit passes on the others. Observed on macos-latest, 3.10; nothing is macOS specific.

Since the suite runs with fail-fast, one occurrence cancels the rest of the matrix, and the cancelled jobs are reported as failures.

Expected behaviour

_resolve_conn_dir should tolerate the directory disappearing underneath it, and the tests should not share a real path under ~/.dstack.

dstack version

master (752de2826)

Additional information

Seen in https://github.com/dstackai/dstack/actions/runs/30932775255/job/92071568686. Re-running the job passed with no change, which is consistent with a race rather than a stale-state problem.