Skip to content

[Cross-SDK] Expose portable worker connection and lifecycle status #795

Open
Open
[Cross-SDK] Expose portable worker connection and lifecycle status#795
Labels
EnhancementNew feature or requestObservabilityTrouble seeing app health or how functions run.

Description

@YunchuWang
wangbill (YunchuWang)
opened on Aug 13, 2026

Summary

Portable Durable Task workers generally report startup success before they have
connected to the backend. Today, customers do not have a supported API for
observing the worker's lifecycle or connection state and must infer it from logs.

This feature request asks for a consistent, public way across the .NET, Python,
Java, and JavaScript/TypeScript SDKs to determine whether a worker is starting,
connected, reconnecting, stopping, stopped, or permanently faulted.

Customer problem

Customers running workers in Kubernetes, Azure Container Apps, worker services,
or other self-hosted environments may want to:

  • expose an application readiness or health endpoint;
  • wait for a worker connection before accepting dependent work;
  • alert when a worker has been disconnected for too long;
  • coordinate shutdown, failover, or custom lifecycle management; and
  • distinguish "the worker loop was launched" from "the backend connection is active."

Parsing log messages is not a stable API, and transport channel state alone does
not prove that the worker's GetWorkItems stream is active.

Desired outcome

Provide at least one supported, typed mechanism for customers to observe worker
lifecycle and backend-connection state, with equivalent semantics across the
portable SDKs.

Existing startup behavior does not need to change. In particular, this request
does not require start() or host startup to block until connected.

Design options to consider

The following are alternative or independently adoptable approaches. This
feature does not require implementing all of them.

  1. Pollable status snapshot
    Expose a current status/state property, potentially with timestamps and the
    last sanitized connection error.

  2. Status transition notifications
    Expose an idiomatic event, listener, callback, observable, or async iterator
    for customers that do not want to poll.

  3. Explicit wait helper
    Add an opt-in WaitUntilConnected/wait_until_connected API with timeout and
    cancellation, while preserving nonblocking startup by default.

  4. Protocol-backed readiness
    If local SDK signals are not strong enough, add an explicit backend
    acknowledgment that the work-item stream has been accepted and registered.
    This would be a larger, separately coordinated protocol change.

Possible common states include NotStarted, Connecting, Connected,
Reconnecting, Stopping, Stopped, and terminal Faulted. Exact naming and
which option provides the best cost/value tradeoff are open for design.

Scope

  • Align semantics across durabletask-dotnet, durabletask-python,
    durabletask-java, and durabletask-js.
  • Avoid requiring customers to parse logs or consume raw gRPC/protobuf types.
  • Clearly distinguish transient reconnecting from a worker loop that has
    permanently terminated.
  • Document what "Connected" or "Ready" proves and how quickly a half-open
    connection is detected.

Detailed cross-SDK findings and design tradeoffs are included in a follow-up
comment.

Related work

Activity

  1. wangbill (YunchuWang)
    added
    EnhancementNew feature or request
    ObservabilityTrouble seeing app health or how functions run.
    on Aug 13, 2026

  2. YunchuWang commented on Aug 13, 2026

    @YunchuWang
    wangbill (YunchuWang)
    on Aug 13, 2026
    MemberAuthor

    Detailed cross-SDK findings

    Research date: 2026-08-13. The links below are pinned to the default-branch
    revisions examined.

    The official portable SDK matrix currently contains .NET, Python, Java, and
    JavaScript/TypeScript:
    https://learn.microsoft.com/azure/durable-task/sdks/durable-task-overview

    Go is also available experimentally for self-hosted scenarios, so it is noted
    separately but should not necessarily be part of the initial conformance scope.

    Current startup and reconnect behavior

    SDK What startup currently means Connection and reconnect behavior Public worker status API
    .NET The worker is a BackgroundService. Generic Host startup schedules its long-running ExecuteAsync; it does not wait for backend connectivity. A bounded Hello handshake is followed by the work-item stream. Connection failures use full-jitter exponential backoff and retry indefinitely until host shutdown. Repeated transport failures can recreate an SDK-owned channel, and silent streams are detected. None
    Python TaskHubGrpcWorker.start() creates a background thread, sets a private running flag, and returns before that thread connects. Uses a bounded Hello, full-jitter exponential reconnect backoff, channel recreation after repeated transport failures, and indefinite retries until stop(). None; _is_running is private
    Java start() launches a background thread and returns. startAndBlock() blocks for the worker lifetime, not merely until the first connection. It opens GetWorkItems without a separate Hello handshake and retries failures every five seconds. None
    JavaScript/TypeScript await worker.start() launches a fire-and-forget loop and returns before Hello or stream readiness. On current main, an immediate cold-start Hello failure terminates the loop while _isRunning remains true; after a stream was established, stream loss uses indefinite exponential reconnect. Open PR #353 adds bounded Hello attempts, cancellable retry, and indefinite cold-start retry while intentionally preserving nonblocking start(). None; _isRunning is private

    Sources:

    Experimental Go behavior

    The experimental Go client is the one exception to the dominant startup model.
    StartWorkItemListener performs its first Hello and GetWorkItems setup
    synchronously. It returns the first connection error to the caller. Once that
    initial setup succeeds, later reconnects happen in a background goroutine with
    indefinite backoff.

    Source:
    https://github.com/microsoft/durabletask-go/blob/9c9e2d6d4cc3609c28bc2cc660ab5311f0217593/client/worker_grpc.go

    Existing signals are internal or log-based

    All four SDKs emit connection or retry log messages, and JavaScript supports a
    structured logger. These can be observed, but they are not a typed lifecycle
    contract and their event/message details are not an appropriate customer state
    API.

    The shared protocol contains:

    Neither is exposed as public worker status. Ping proves backend reachability,
    not that this worker's work-item stream is registered. HealthPing is consumed
    internally as stream traffic.

    The distinction matters in JavaScript today: the "worker connected" log is
    written after Hello succeeds and a lazy gRPC stream object is created, before
    response metadata, a health ping, or a work item is received:
    https://github.com/microsoft/durabletask-js/blob/0d132c59431bba8535857818ab26653f7874fc1d/packages/durabletask-js/src/worker/task-hub-grpc-worker.ts#L405-L420

    Design tradeoffs

    These are options to evaluate, not cumulative requirements.

    Option A: pollable status snapshot

    Example data could include state, state-change time, last-connected time, last
    successful backend contact, advisory retry attempt, and a sanitized SDK-owned
    error code/message.

    Pros:

    • Small public surface and relatively low implementation cost.
    • Directly supports synchronous readiness/health endpoints.
    • Does not require a backend or protocol change.

    Cons:

    • Customers must poll and choose their own interval.
    • Short transitions can be missed.
    • Every customer needing startup coordination must implement timeout and race
      handling.

    Option B: transition subscription

    Each language could use its idiomatic mechanism: .NET event/observable/async
    enumerable, Python callback or iterator, Java listener, and JavaScript event or
    async iterator.

    Pros:

    • Good for alerting and explicit lifecycle management.
    • Avoids wasteful polling and preserves transition context.

    Cons:

    • More concurrency and disposal semantics must be specified.
    • Slow or throwing subscribers must never block or terminate the worker loop.
    • A replay-latest rule is needed so subscribing after start() does not miss
      the current state.

    Option C: explicit wait-until-connected helper

    Keep startup nonblocking and provide an opt-in wait with required timeout and
    cancellation.

    Pros:

    • Directly solves startup coordination.
    • Avoids changing established host/start behavior.
    • Centralizes read/subscribe race handling inside the SDK.

    Cons:

    • Requires a precise definition of "Connected."
    • Does not by itself support ongoing alerting unless paired with polling or
      notifications.

    Option D: protocol-backed readiness acknowledgment

    The backend explicitly acknowledges that GetWorkItems was accepted and the
    worker was registered, or the protocol guarantees an immediate equivalent
    signal.

    Pros:

    • Provides the strongest and most consistent readiness definition.
    • Avoids treating Hello or lazy stream creation as worker readiness.

    Cons:

    • Requires protobuf and backend changes plus capability/version negotiation.
    • Older backends need a documented fallback.
    • Requires coordinated rollout across more repositories.

    State-model considerations

    If a state enum is selected, one possible model is:

    • NotStarted: start has never been called.
    • Connecting: start was called, but this run has never connected.
    • Connected: the strongest agreed work-stream acceptance signal was observed.
    • Reconnecting: this run was connected previously and retries are active.
    • Stopping: shutdown is in progress.
    • Stopped: shutdown completed and no attempts remain.
    • Faulted: the worker loop terminated due to an unrecoverable error.

    Faulted should not represent ordinary connection failures when the worker is
    still retrying. Alerting for a prolonged outage would instead use
    Connecting/Reconnecting plus elapsed time and last successful contact.

    Important semantic questions:

    • Does Connected mean Hello succeeded, response metadata arrived, or the
      first health/work message arrived?
    • How stale can Connected become during a half-open connection?
    • Are transitions published only when the state changes, or on every retry?
    • Does a subscriber immediately receive the current snapshot?
    • How are subscriber exceptions, ordering, unsubscription, and backpressure
      handled?
    • Does waiting fail promptly when the worker becomes Stopped or Faulted?

    Related evidence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    EnhancementNew feature or requestObservabilityTrouble seeing app health or how functions run.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions