Skip to content

feat(functions): add opt-in durable HTTP failure retries - #377

Merged
wangbill (YunchuWang) merged 2 commits into
mainfrom
yunchuwang-durable-http-retries
Oct 2, 2026
Merged

wangbill (YunchuWang) merged 2 commits into
mainfrom
yunchuwang-durable-http-retries

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

What changed?

  • Add HttpRetryOptions and callHttp({ retryOptions }), snapshotting the policy and status list into the durable request.
  • Reuse the core activity retry engine and durable timers for selected HTTP statuses and activity/transport failures. No second retry engine or core changes.
  • Classify retryable status codes before buffering the response. Cancel the discarded response body before failing the activity, so a response whose body never ends cannot block the retry. Returned responses still include their complete body.
const retryOptions = new df.HttpRetryOptions(1000, 4);
retryOptions.backoffCoefficient = 2;
retryOptions.maxRetryIntervalInMilliseconds = 10000;
retryOptions.retryTimeoutInMilliseconds = 60000;
retryOptions.statusCodesToRetry = [429, 500, 503];
const response = yield context.df.callHttp({ method: "GET", url, retryOptions });

Why is this change needed?

Previously callHttp returned error status responses without durable failure retries. With explicit options it now retries selected responses, returns excluded statuses, and throws TaskFailedError on exhaustion rather than returning the last failed response.

The reference is the .NET in-process Durable Functions extension, not the generic .NET SDK:

  • HttpRetryOptions: backoff 1, six-day interval cap, unlimited retry timeout by default.
  • TaskHttpActivityShim: omitted/empty status lists use non-2xx semantics (including 4xx/5xx and unfollowed 3xx); explicit lists are exact, including 202.
  • CreateLocationPollRequest does not propagate retry options. This PR matches that boundary: initial requests retry; subsequent 202 Location polls do not. Initial 503 -> 202 -> poll 503 therefore returns the poll's 503 without retrying it.

Omitting options preserves current behavior. Existing polling, redirects, credential stripping, and URL validation are unchanged. README documents JavaScript millisecond/-1 conventions, core retry-timeout boundaries, idempotency considerations, and versioning/draining in-flight instances before enabling retries.

Issues / work items


Project checklist

  • Release notes are not required for the next release
    • Otherwise: Notes added to packages/azure-functions-durable/CHANGELOG.md
  • Backport is not required
    • Otherwise: Backport tracked by issue/PR (not assessed)
  • All required tests have been added/updated (unit tests, E2E tests)
  • Breaking change?
    • If yes:
      • Impact: No behavior change without explicit retry options.
      • Migration guidance: Version or drain in-flight instances before changing their retry behavior.

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No
  • Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
  • Yes, an AI agent generated most of this PR

If AI was used:

  • Tool(s): GitHub Copilot.
  • AI-assisted areas/files: HTTP retry implementation, tests, README, changelog.
  • What you changed after AI output: Agent iterations corrected test-harness issues, added runtime null-list coverage, and type-checked the README example. The review follow-up fixed response-body ordering, added bounded cancellation regressions, restored two unrelated formatting hunks, and validated the immutable candidate against real Azure DTS. Human review remains pending.

AI verification (required if AI was used):

  • I understand the code and can explain it
  • I verified referenced APIs/types exist and are correct
  • I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
  • I reviewed concurrency/async behavior
  • I checked for unintended breaking or behavior changes

Testing

Validated commit: 57efe8bda358ad450a9b92af2f4b8f3a519df379, tree 800a2a6d1d918a3cbd040956d122b169a304087a. This exact commit was tested before the ordinary follow-up push.

Automated tests

  • Passed: 113 tests across seven scoped HTTP/context/retry/worker suites, with real loopback HTTP and both protobuf replay and in-memory execution.
  • The new bounded open-body test first failed on the previous implementation: a retryable 503 with an unfinished body blocked until the explicit two-second deadline. Three cancellation assertions also failed before the fix. The final run passed, with owned sockets closed in cleanup.
  • Passed: npm run build -w durable-functions (includes core build), npm run build -w @microsoft/durabletask-js-azuremanaged, scoped ESLint, changed-range/README formatting checks, git diff --check, and normal Husky/lint-staged commit checks. Existing baseline formatting was retained rather than rewriting unrelated code.
  • The unchanged README code example was also strict-TypeScript checked during initial implementation.
  • Coverage includes retry/success/exhaustion, excluded/default/explicit-202 statuses, transport errors, snapshot validation, backoff/cap/timeout boundaries, six-day timer segmentation, replay, caller-visible exceptions, and polling interaction. Follow-up cases cover cancellation before buffering, null bodies, surfaced cancellation errors, and intact returned content.

Real Azure validation

2026-10-02, first and only live run: 6/6 scenarios passed. Windows / Node.js 24.14.0; a new, isolated Consumption task hub in westus2, with hub-scoped data access and a restricted IP allowlist.

The local Azure-managed SDK worker executed the public Functions callHttp wrapper and unchanged built-in HTTP activity/poll orchestrator against production Azure DTS. Authenticated TLS Hello and a missing-instance read passed first. Scheduling, dispatch, persistence, failures, retry timers, and history reads used the real service; no generated work items or synthetic timer events were used in this run.

Scenario HTTP calls Persisted TaskFailed TimerCreated / TimerFired Result
503 with a body the server never ends, then 200 2 1 1 / 1 Body cancellation observed before the second POST; completed with 200
429 -> 503 -> 200 3 2 2 / 2 Completed with 200
Exhausted 503 retries 2 2 1 / 1 HTTP child failed; caller caught TaskFailedError
503 excluded from selected statuses 1 0 0 / 0 Returned 503 and its complete body
No retry options, 503 1 0 0 / 0 Returned 503 and its complete body
Initial 503 -> 202 -> Location GET 503 3 1 2 / 2 Returned poll 503 without failure retry; initial retry and polling timers persisted

Totals: 12 HTTP requests, 6 TaskFailed events, 6 TimerCreated and 6 TimerFired events. Root replay observations and exact HTTP request counts passed. The polling request changed POST to GET, dropped the request body and x-functions-key, retained the same-origin fixture authorization header, and carried no retry policy.

The stalled-body server recorded the first POST at 15:23:47.034Z, client cancellation without a server-side end at 15:23:47.046Z, and the next POST at 15:23:47.326Z.

Evidence integrity: candidate HEAD was unchanged before/after execution; harness SHA-256 was unchanged at 3b845b47dcc86cce408bdb0491b5c52cb42bc8c81ab42a93a9be2f16ce0f0195; source and compiled module hashes were recorded and independently read back. Raw histories and cleanup evidence are retained locally, not published with private environment identifiers.

Cleanup: all 12 exact root/child instance IDs were individually purged and verified absent; the worker, client, and HTTP server stopped. The temporary role assignment and task hub were separately verified absent, and the scheduler/resource group were removed with az group exists returning false. No test infrastructure remains.

Scope limits: HTTP responses came from a controlled loopback server. This was not an Azure Functions-host deployment, cloud HTTP endpoint, load test, or simulated Azure service outage. Import-time Functions registration was in test mode; the actual HTTP built-ins were explicitly registered in the real Azure-managed worker. Local Functions-host/Azurite E2E and full repository suites were not run for this follow-up. Hosted CI is tracked separately and is not substituted for the live feature-specific evidence above.


Notes for reviewers

  • The initial-request-only retry boundary is intentional source parity, not per-poll retry support. This is narrower than treating every 202 hop as a fresh retry budget.
  • Response-body review follow-up.
  • Human verification checkboxes remain unchecked.

Add HttpRetryOptions and serialize its policy and status-code snapshot for the built-in HTTP activity. Reuse durable RetryPolicy execution, preserve calls without options, and match .NET in-process initial-request-only retry semantics.

Closes #367

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e0af01a5-0dfa-4e71-a660-c4186e65d7e0
Copilot AI balanced review requested due to automatic review settings September 29, 2026 16:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Retriable responses are fully buffered before status evaluation, which can delay or prevent durable retries for large or non-terminating bodies.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Adds opt-in durable HTTP retries to classic callHttp, reusing core activity retries and preserving separate 202 polling behavior.

Changes:

  • Adds and exports HttpRetryOptions.
  • Applies durable retries to configured HTTP statuses and transport failures.
  • Documents behavior and adds replay, timeout, polling, and exhaustion tests.
File Description
src/​http/​models.ts Defines HTTP retry options and payload fields.
src/​http/​builtin.ts Applies retry policies to initial HTTP requests.
src/​orchestration-context.ts Snapshots retry configuration into durable requests.
src/​index.ts Exports HttpRetryOptions.
test/​unit/​http-retry.spec.ts Tests HTTP retry execution and replay.
test/​unit/​orchestration-context.spec.ts Tests policy validation and serialization.
README.md Documents usage and semantics.
CHANGELOG.md Records the feature.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/azure-functions-durable/src/http/builtin.ts Outdated
Classify selected HTTP failures as soon as headers arrive and cancel their unused body so an open response cannot block durable retries. Keep returned response bodies intact and restore unrelated baseline formatting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e0af01a5-0dfa-4e71-a660-c4186e65d7e0
@YunchuWang
wangbill (YunchuWang) merged commit 72b79a5 into main Oct 2, 2026
31 checks passed
@YunchuWang
wangbill (YunchuWang) deleted the yunchuwang-durable-http-retries branch October 2, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HTTP-specific retry options to durable-functions v4 callHttp

3 participants