Skip to content

feat: [3/6] add interoperable Azure Blob payload storage - #390

Open
wangbill (YunchuWang) wants to merge 1 commit into
yunchuwang-payload-phase-2-sdkfrom
yunchuwang-payload-phase-3-blob
Open

wangbill (YunchuWang) wants to merge 1 commit into
yunchuwang-payload-phase-2-sdkfrom
yunchuwang-payload-phase-3-blob

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

What changed?

  • Add the optional Azure Blob provider package with credential-free v2 writes and legacy v1 reads, UTF-8/gzip representation, private container initialization and ownership metadata.
  • Preserve credentials/pipelines across same-service containers and path-style Azurite URLs; require identity for cross-account routing instead of forwarding configured account keys/SAS.
  • Retain single-flight initialization cancellation/generation recovery and explicit ownership/ETag/snapshot deletion safety.
  • Carry 39 offline Azure SDK/provider tests, seven actual .NET gzip vectors and 10 opt-in real Azurite tests. Add only the package workspace lock entry and exact Jest/TypeScript mappings.

Why is this change needed?

  • Phase 3/6 of the requested large-payload implementation split. This PR's base is phase 2's branch, so its diff excludes previously reviewed phases.
  • Behavioral reference: durabletask-dotnet 91739687. The complete stack preserves the original implementation at c9b53fcb3ab8585c4b3b22f9c0500cc9dd122c5a, with the documented instance-kind routing correction.

Issues / work items

  • Resolves: N/A (user-requested phased feature review).
  • Related: original oversized PR #387. It is preserved while the coordinator verifies this replacement stack.

Project checklist

  • Release notes are not required for the next release
    • Otherwise: Notes added to CHANGELOG.md
  • Backport is not required
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • All required tests have been added/updated (unit tests, E2E tests)
  • Breaking change?
    • If yes:
      • Impact: No intentional API break in this phase; the payload feature remains opt-in.
      • Migration guidance: Do not release partial phases. Merge the stack in order and publish the completed core before dependent packages.

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No
  • Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
  • Yes, an AI agent generated most of this PR

If AI was used:

  • Tool(s): GitHub Copilot.
  • AI-assisted areas/files: This phase's source, regression tests, and directly related metadata/documentation.
  • What you changed after AI output: Split the implementation along independently buildable boundaries; retained focused regression evidence and removed unrelated formatting-only changes. Human review is still required.

AI verification (required if AI was used):

  • I understand the code and can explain it
  • I verified referenced APIs/types exist and are correct
  • I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
  • I reviewed concurrency/async behavior
  • I checked for unintended breaking or behavior changes

These human attestations are intentionally left unchecked.


Testing

Automated tests

  • Result: Passed locally for this phase, without later-phase files.
  • npm ci --offline --no-audit --no-fund — passed; no existing dependency upgrades.
  • npm run build — passed all five workspaces with no later cleanup code.
  • npm run lint — passed.
  • npm run test:unit -w @microsoft/durabletask-js-azureblob — passed 39 provider tests; 10 optional service tests skipped in that command.
  • npm run test:azurite -w @microsoft/durabletask-js-azureblob with a local-only test connection string — passed all 10 tests against isolated Azurite 3.37.0.
  • prettier --check/formatting of new package files and incremental diff hygiene — passed.
  • git diff --check yunchuwang-payload-phase-2-sdk...HEAD — passed.
  • Hosted CI: pending/not inferred. Existing main-base filters mean some workflows will not run for stacked PRs; no workflow triggers were broadened.

Manual validation (only if runtime/behavior changed)

  • Environment (OS, Node.js version, components): Windows, Node.js v24.14.0; Azure Storage Blob SDK 12.31.0; offline real-SDK HTTP pipeline and isolated loopback Azurite 3.37.0.
  • Steps + observed results:
    1. Materialized this phase on its declared predecessor, with no future-phase imports.
    2. Ran the exact owning checks above and inspected the incremental patch.
    3. Verified .NET-produced compressed bytes, UTF-8 and uncompressed round trips, token routing and owned/unowned blob/snapshot deletion against real Azurite.
  • Evidence (optional): Full local logs are session artifacts, not repository changes.

Notes for reviewers

Merge order Phase PR Base branch
1 Core contracts #388 main
2 SDK integration #389 yunchuwang-payload-phase-1-core
3 Azure Blob provider #390 yunchuwang-payload-phase-2-sdk
4 Purge protocol/client #391 yunchuwang-payload-phase-3-blob
5 Purge worker #392 yunchuwang-payload-phase-4-client
6 Release/usage/consumers #393 yunchuwang-payload-phase-5-worker

Merge 1 → 2 → 3 → 4 → 5 → 6. Each PR targets its predecessor; retarget the next PR only after its prerequisite merges. Do not publish packages or describe a partial phase as the complete feature.

  • The provider supports explicit safe deletion, but does not register or expose scheduler-driven auto-purge in this phase.
  • Provider documentation describes only APIs present in phases 1–3; client cleanup protocol and worker runner follow separately.
  • Core >=0.4.1 remains mandatory; package installation metadata must not accept published 0.4.0.
  • Full live DTS/mixed-.NET execution has not been run locally (Docker daemon unavailable). The final phase records aggregate local Blob/Azurite, source/wire interoperability and packed-consumer evidence without claiming absent hosted checks.

Phase 3 of 6: add optional UTF-8/gzip storage, credential-safe v1/v2 routing, cancellation-aware single-flight container initialization and conditional owned-blob deletion. Preserve .NET fixtures and real Azurite tests without advertising the later cleanup lifecycle.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81e20512-d3a2-4f8e-a574-f396140ec9c7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant