Skip to content

feat(client)!: report why the client could not connect to the engine - #677

Merged
Benehiko merged 3 commits into
mainfrom
feat/client-engine-not-running
Oct 8, 2026
Merged

Benehiko merged 3 commits into
mainfrom
feat/client-engine-not-running

Conversation

@Benehiko

@Benehiko Benehiko commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Why

Every failure to dial the engine socket was wrapped in ErrSecretsEngineNotAvailable. A missing socket, a stale one, permission denied and a dial timeout all looked the same. Clients that try the engine before falling back need to tell those apart: "not running" is routine and only worth a debug log, while "permission denied" or "timed out" means something is wrong. This came up in review of docker/docker-agent#4535, which currently has to dig the errno out of the wrapped error itself.

What

Breaking: ErrSecretsEngineNotAvailable is removed. It is replaced by one error type that says why the connection failed:

type ConnectError struct {
	Reason     ConnectReason // ReasonUnknown, ReasonNotRunning, ReasonPermissionDenied, ReasonTimeout
	SocketPath string        // empty with WithDialContext
	Err        error         // the underlying dial error, still inspectable
}

Each reason except ReasonUnknown has a sentinel that errors.Is matches: ErrSecretsEngineNotRunning, ErrSecretsEnginePermissionDenied, ErrSecretsEngineTimeout.

  • Not running: the socket does not exist (fs.ErrNotExist), or nothing listens on it (ECONNREFUSED, or WSAECONNREFUSED on Windows).
  • Permission denied: fs.ErrPermission.
  • Timeout: context.DeadlineExceeded, or a net.Error whose Timeout() is true.
  • Unknown: anything else. It doesn't match any sentinel, so an unexpected errno isn't mistaken for a stopped engine.
  • Cancelled: a context.Canceled dial is returned unchanged. The caller cancelled it, so it isn't reported as an engine failure.
  • Message: includes the socket path, e.g. secrets engine is not running at /run/secrets-engine.sock: dial unix ...: connect: no such file or directory.

The removal is deliberate. Keeping the old name with a narrower meaning would still compile but quietly change behaviour, whereas a build failure makes each caller pick the case it means.

Using it

Fall back on any connection failure (e.g. when the engine is optional):

envelopes, err := c.GetSecrets(ctx, pattern)
if _, ok := errors.AsType[*client.ConnectError](err); ok {
	log.Debug("secrets engine unavailable, falling back", "err", err)
	return fallback(ctx, pattern)
}
if err != nil {
	return err
}

Treat "not running" as routine and surface everything else:

switch {
case errors.Is(err, client.ErrSecretsEngineNotRunning):
	log.Debug("secrets engine not running")
	return fallback(ctx, pattern)
case errors.Is(err, client.ErrSecretsEnginePermissionDenied):
	return fmt.Errorf("check permissions on the secrets engine socket: %w", err)
}

Branch on the reason, and use the socket path in a message:

if ce, ok := errors.AsType[*client.ConnectError](err); ok {
	switch ce.Reason {
	case client.ReasonNotRunning:
		fmt.Fprintf(os.Stderr, "start the secrets engine (looked at %s)\n", ce.SocketPath)
	case client.ReasonTimeout:
		fmt.Fprintln(os.Stderr, "the secrets engine did not respond; is it hung?")
	default:
		fmt.Fprintln(os.Stderr, err)
	}
}

Migrating

Before After
errors.Is(err, client.ErrSecretsEngineNotAvailable) errors.As / errors.AsType[*client.ConnectError](err) for any connection failure
digging syscall.ENOENT / ECONNREFUSED out of the error errors.Is(err, client.ErrSecretsEngineNotRunning)

errors.AsType needs Go 1.26, which this module already requires. errors.As with a *client.ConnectError target works too.

pass

preflightPing no longer wraps every error as "not available":

  • An engine that accepts the connection but doesn't answer within the preflight timeout becomes a ConnectError with ReasonTimeout.
  • Only the ping's own timeout counts. The ping context is created with context.WithTimeoutCause, and the error is wrapped only when context.Cause reports that timeout. If the caller's context expires first, its context.DeadlineExceeded is returned unchanged rather than reported as a hung engine.
  • ConnectErrors already classified by the client, and real errors returned by the engine, pass through unchanged.

Fixed a test helper: deadSocket() built its path from t.TempDir(), which on macOS is longer than the Unix socket path limit (~104 bytes). The tests were failing with EINVAL, not "no such file", and only passed because every failure counted as "not available". It now uses testhelper.RandomShortSocketName().

Windows

Tested on a Windows 11 VM (build 26100, ARM64). Test binaries were cross-compiled with GOOS=windows GOARCH=arm64 go test -c and run on the VM:

Suite Result
client PASS
client/dockerhub PASS
plugins/pass/commands PASS
  • TestConnectErrorFromSocket dials real AF_UNIX sockets. A missing socket and a stale socket (nothing listening) are both classified ReasonNotRunning. That confirms Windows reports a missing socket as fs.ErrNotExist and a refused connection as WSAECONNREFUSED (10061). See the inline comment on errno_windows.go.
  • Permission denied: a probe that added a deny-all ACL (icacls <sock> /deny %USERNAME%:(F)) for the current user still connected successfully. So on Windows, at least in that setup, ReasonPermissionDenied does not occur; it is effectively Unix-only. Other users or ACL setups were not tested. The end-to-end permission test is skipped on Windows.

Windows client tests still aren't in CI (only keychain.yml runs on Windows).

Tests

  • TestConnectError: a table test of the real error shape (*net.OpError → *os.SyscallError → errno). Each case checks the reason, the socket path, that only the matching sentinel matches, and that the dial error is still inspectable. Also covers cancelled contexts and the message format.
  • TestConnectErrorFromSocket: end to end through GetSecrets against a missing socket, a stale socket, and a chmod 000 socket (the last is skipped on Windows and as root).
  • TestPreflightPing: not running, a non-connection engine error, a hung engine (→ ErrSecretsEngineTimeout), and a caller deadline that fires before the ping timeout (→ context.DeadlineExceeded, not ErrSecretsEngineTimeout).
  • go vet (darwin, windows) and golangci-lint are clean for client and plugins/pass. credentialhelper, plugin and store build.

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟡 NEEDS ATTENTION

One medium-severity finding on the Windows errno classification path.

Comment thread client/errno_windows.go
@Benehiko Benehiko changed the title feat(client): tell a stopped engine apart from other dial failures feat(client): tell an unreachable engine apart from a missing one Oct 8, 2026
@Benehiko
Benehiko force-pushed the feat/client-engine-not-running branch 3 times, most recently from f31eb68 to f44b2d6 Compare October 8, 2026 07:27
@Benehiko Benehiko changed the title feat(client): tell an unreachable engine apart from a missing one feat(client)!: report why the client could not connect to the engine Oct 8, 2026
Comment thread client/errno_windows.go
import "syscall"

// errConnRefused is WSAECONNREFUSED; Winsock never returns syscall.ECONNREFUSED.
const errConnRefused = syscall.Errno(10061)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on Windows: a Windows 11 VM (build 26100, ARM64), running the test binaries cross-compiled with GOOS=windows GOARCH=arm64 go test -c.

TestConnectErrorFromSocket dials real AF_UNIX sockets:

  • Stale socket (SetUnlinkOnClose(false), then close the listener): Winsock returns WSAECONNREFUSED (10061), so this constant matches and the error is ReasonNotRunning. syscall.ECONNREFUSED would not have matched.
  • Missing socket: reported as fs.ErrNotExist via syscall.Errno.Is, also ReasonNotRunning.

The full client, client/dockerhub and plugins/pass/commands suites pass on the VM.

Not covered here: a deny-all ACL on the socket for the current user did not stop the connection, so ReasonPermissionDenied does not show up on Windows in that setup.

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟡 NEEDS ATTENTION

Comment thread plugins/pass/commands/client.go
Benehiko added a commit to Benehiko/docker-agent that referenced this pull request Oct 8, 2026
The SDK wraps every failed dial in ErrSecretsEngineNotAvailable,
permission denied and timeouts included, so an engine we may not connect
to was logged at debug level as if Docker Desktop were not installed.
Only a missing socket or one nothing listens on is routine now; anything
else gets the warning. Tests build the SDK's real error shape instead of
wrapping a string. docker/secrets-engine#677 adds
ErrSecretsEngineUnreachable to do this in the SDK.
@Benehiko
Benehiko force-pushed the feat/client-engine-not-running branch 2 times, most recently from 2ae84e2 to 74d43ee Compare October 8, 2026 08:53
@Benehiko
Benehiko requested a review from joe0BAB October 8, 2026 08:56
Comment thread client/errors.go Outdated
Comment thread client/errors.go Outdated
Comment thread client/errors.go Outdated
Comment thread client/errors.go Outdated
Comment thread client/errors.go Outdated
Every dial failure was wrapped in ErrSecretsEngineNotAvailable, so a caller could not tell an engine that is not running from one it may not connect to. Replace it with ConnectError, which carries a Reason (NotRunning, PermissionDenied, Timeout or Unknown), the socket path and the underlying dial error. Match any connection failure with errors.As, or one reason with errors.Is and ErrSecretsEngineNotRunning, ErrSecretsEnginePermissionDenied or ErrSecretsEngineTimeout. A cancelled context is returned unchanged rather than reported as an engine failure.

pass: preflightPing reports an engine that accepts the connection but does not answer as a ReasonTimeout ConnectError, and passes other engine errors through. deadSocket now uses a short path; the t.TempDir() path exceeded the Unix socket path limit on macOS and failed with EINVAL.

BREAKING CHANGE: ErrSecretsEngineNotAvailable is removed. Use errors.As with *ConnectError, or errors.Is with one of the new sentinels.

Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
preflightPing labelled any context.DeadlineExceeded as a ReasonTimeout ConnectError, including when the caller's own context expired before the ping timeout. Record errPingTimeout as the cause of the ping's own deadline and only wrap the error when context.Cause reports it; otherwise return the caller's error unchanged. ConnectErrors already classified by the client are returned as is.

Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
@Benehiko
Benehiko force-pushed the feat/client-engine-not-running branch 2 times, most recently from 7cb0d3e to 04a51ec Compare October 8, 2026 12:27
@Benehiko
Benehiko enabled auto-merge October 8, 2026 12:30
Comment thread client/client.go Outdated
Address review feedback by dropping doc comments that restate the code in client/errors.go and plugins/pass/commands/client.go.

Also fold the package-level connectError helper into the client.connectError method, its only caller. No behaviour change.

Signed-off-by: Alano Terblanche <18033717+Benehiko@users.noreply.github.com>
@Benehiko
Benehiko force-pushed the feat/client-engine-not-running branch from 04a51ec to fadde1c Compare October 8, 2026 12:33
@Benehiko
Benehiko merged commit c2a0811 into main Oct 8, 2026
12 checks passed
@Benehiko
Benehiko deleted the feat/client-engine-not-running branch October 8, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants