Skip to content

fix: support Docker daemon over SSH (DOCKER_HOST=ssh://...)#3078

Open
lucklove wants to merge 3 commits into
replicate:mainfrom
lucklove:main
Open

fix: support Docker daemon over SSH (DOCKER_HOST=ssh://...)#3078
lucklove wants to merge 3 commits into
replicate:mainfrom
lucklove:main

Conversation

@lucklove

Copy link
Copy Markdown

Problem

When DOCKER_HOST or the active Docker context is set to an SSH endpoint (e.g. ssh://user@host), cog fails with:

error pinging docker daemon: error during connect: Head "http://user@host/_ping": dial tcp: lookup user@host: no such host

The root cause is that the Docker Go SDK's client.WithHost() does not understand the ssh:// scheme — it attempts a plain TCP/HTTP connection instead of tunneling over SSH.

The standard docker CLI handles this transparently via connhelper.GetConnectionHelper(), which returns an SSH-based dialer for ssh:// hosts.

Fix

After building the dockerClientOpts slice, call connhelper.GetConnectionHelper(host). If the host is an SSH URL, inject the returned SSH dialer via client.WithDialContext. This mirrors how the official Docker CLI handles remote SSH daemons.

if helper, err := connhelper.GetConnectionHelper(clientOptions.host); err == nil && helper != nil {
    dockerClientOpts = append(dockerClientOpts, client.WithDialContext(helper.Dialer))
}

github.com/docker/cli is already a dependency, so no new dependencies are introduced.

Testing

  • Existing pkg/docker tests all pass
  • Manually verified cog build works against a remote Docker daemon via DOCKER_HOST=ssh://root@<host>

@lucklove
lucklove requested a review from a team as a code owner June 27, 2026 11:58

@anish-sahoo anish-sahoo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for putting this change together, we need some adjustments before we can move forward. Please configure the Docker client completely from the SSH connection helper and add coverage for the affected environment combinations.

Comment thread pkg/docker/docker.go
@@ -62,6 +63,10 @@ func NewClient(ctx context.Context, opts ...Option) (*apiClient, error) {
client.WithHost(clientOptions.host),
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will break whenever DOCKER_CERT_PATH is set. WithTLSClientConfigFromEnv() has already installed TLS before this branch, and adding only the helper dialer preserves that configuration. The client then attempts a TLS handshake over docker system dial-stdio, which speaks plain Docker HTTP. Replace the transport with a non-TLS transport for the SSH path.

Comment thread pkg/docker/docker.go
@@ -62,6 +63,10 @@ func NewClient(ctx context.Context, opts ...Option) (*apiClient, error) {
client.WithHost(clientOptions.host),
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The helper returns Host specifically as the valid dummy HTTP origin for requests over its dialer, but it is never applied. Retaining ssh://user@host leaves an invalid HTTP Host, and a socket-path SSH URL has the same issue. This is especially visible when HTTP_PROXY is set, where Go rejects the invalid Host before opening SSH. Apply client.WithHost(helper.Host) and ensure proxy settings do not carry into connection-helper transports.

Comment thread pkg/docker/docker.go
@@ -62,6 +63,10 @@ func NewClient(ctx context.Context, opts ...Option) (*apiClient, error) {
client.WithHost(clientOptions.host),
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please return this error instead of ignoring it. For an invalid SSH endpoint such as ssh://user:password@host or ssh://host?foo=bar, connhelper provides clear validation; ignoring it instead falls back to a malformed Docker client and surfaces an unrelated connection failure.

Configure the docker client entirely from connhelper for ssh:// hosts:
- use helper.Host as the HTTP origin
- install a non-TLS, proxy-disabled transport so DOCKER_CERT_PATH /
  HTTP_PROXY do not leak into the SSH path
- return connhelper errors (invalid ssh URL) instead of silently
  falling through

Covered by pkg/docker/docker_ssh_test.go:
- invalid ssh URL (password / query) returns wrapped error
- happy path pulls alpine:latest over DOCKER_HOST=ssh://root@kr
- DOCKER_CERT_PATH + ssh host no longer breaks
- HTTP_PROXY + ssh host no longer breaks
@lucklove

lucklove commented Jul 18, 2026

Copy link
Copy Markdown
Author

Pushed an updated revision that addresses all three inline comments.

What changed in pkg/docker/docker.go:

  • TLS is no longer applied on the SSH path. When connhelper returns a helper, the client opts only include WithHTTPClient(&http.Client{Transport: &http.Transport{Proxy: nil}}), WithHost(helper.Host), and WithDialContext(helper.Dialer). WithTLSClientConfigFromEnv() is now exclusively on the non-SSH branch, so DOCKER_CERT_PATH can no longer wrap the plain docker system dial-stdio stream in TLS.
  • helper.Host is now the HTTP origin. No more ssh://user@host in the Host header.
  • Transport.Proxy = nil so HTTP_PROXY / HTTPS_PROXY don't try to CONNECT the dummy host before the SSH dialer opens a session.
  • connhelper.GetConnectionHelper errors are returned, wrapped as invalid docker host %q: %w. No more silent fallback for things like ssh://user:password@host or ssh://host?foo=bar.

Coverage (pkg/docker/docker_ssh_test.go, new file):

  • invalid SSH URL surfaces validation error — pure unit test, runs in -short.
  • happy path — pulls alpine:latest over an ssh:// host.
  • ignores DOCKER_CERT_PATH — pulls with DOCKER_TLS_VERIFY=1 + bogus DOCKER_CERT_PATH.
  • ignores HTTP_PROXY — pulls with HTTP_PROXY=http://127.0.0.1:1 etc.

The live subtests are gated by COG_SSH_E2E=1 + COG_SSH_E2E_HOST=<ssh-url>. All four passed end-to-end against a real Docker daemon over SSH:

--- PASS: TestNewClientSSHHost (22.22s)
    --- PASS: invalid_SSH_URL_surfaces_validation_error (0.00s)
    --- PASS: happy_path (6.70s)
    --- PASS: ignores_DOCKER_CERT_PATH (7.15s)
    --- PASS: ignores_HTTP_PROXY (8.37s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants