Repository navigation
feat: add .NET-aligned large payload storage and auto-purge - #387
Closed
wangbill (YunchuWang) wants to merge 3 commits into
Closed
wangbill (YunchuWang) wants to merge 3 commits into
wangbill (YunchuWang) wants to merge 3 commits into
Conversation
Add opt-in payload externalization and hydration across client, worker, history, and entity transports, an optional Azure Blob store, and the DTS auto-purge lifecycle aligned with durabletask-dotnet 91739687ff647699079cdadd627daa00562982b7. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81e20512-d3a2-4f8e-a574-f396140ec9c7
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.SetLargePayloadAutoPurgeRequest.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.SetLargePayloadAutoPurgeResponse.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.SetLargePayloadAutoPurgeResponse.serializeBinaryToWriter = function(message, writer) { | ||
| var f = undefined; |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.LargePayloadTombstone.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.LargePayloadPurgeResult.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.GetLargePayloadTombstonesRequest.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.GetLargePayloadTombstonesResponse.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.ReportLargePayloadPurgeResultsRequest.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.ReportLargePayloadPurgeResultsResponse.toObject = function(includeInstance, msg) { | ||
| var f, obj = { |
| * @suppress {unusedLocalVariables} f is only used for nested messages | ||
| */ | ||
| proto.microsoft.durabletask.largepayloads.ReportLargePayloadPurgeResultsResponse.serializeBinaryToWriter = function(message, writer) { | ||
| var f = undefined; |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Carryover payload externalization and the extension’s minimum core dependency need correction before approval.
2 open findings
What changed in this PR
Adds opt-in external payload storage and automatic cleanup to the Durable Task SDK, with an optional Azure Blob package that keeps storage dependencies out of core.
Changes:
- Integrates payload externalization and hydration into clients, workers, and Azure-managed builders.
- Adds .NET-compatible Blob storage and the DTS auto-purge lifecycle.
- Adds regression tests, interoperability fixtures, documentation, a sample, and release wiring.
| File | Description |
|---|---|
| tsconfig.base.json | Adds Azure Blob package mapping. |
| tools/generate-grpc-javascript.sh | Generates purge bindings and normalizes whitespace. |
| README.md | Introduces payload configuration. |
| packages/durabletask-js/test/payloads.spec.ts | Tests payload processing and bounded concurrency. |
| packages/durabletask-js/test/payload-transport.spec.ts | Tests gRPC payload integration. |
| packages/durabletask-js/test/payload-purge.spec.ts | Tests cleanup lifecycle and runner behavior. |
| packages/durabletask-js/test/client-purge.spec.ts | Tests payload-enabled purge routing. |
| packages/durabletask-js/src/worker/task-hub-grpc-worker.ts | Integrates payload processing and purge tasks. |
| packages/durabletask-js/src/proto/large_payload_purge_pb.d.ts | Adds generated purge message types. |
| packages/durabletask-js/src/proto/large_payload_purge_grpc_pb.js | Adds generated purge RPC bindings. |
| packages/durabletask-js/src/proto/large_payload_purge_grpc_pb.d.ts | Adds generated purge service types. |
| packages/durabletask-js/src/payloads/run-bounded.ts | Bounds concurrent storage operations. |
| packages/durabletask-js/src/payloads/purge.ts | Implements the durable cleanup runner. |
| packages/durabletask-js/src/payloads/purge-worker.ts | Registers cleanup activities and classifies outcomes. |
| packages/durabletask-js/src/payloads/purge-transport.ts | Reuses authenticated transport for purge RPCs. |
| packages/durabletask-js/src/payloads/payload-store.ts | Defines storage contracts and options. |
| packages/durabletask-js/src/payloads/payload-processor.ts | Maps protobuf payload fields for storage. |
| packages/durabletask-js/src/payloads/history-stream.ts | Hydrates streamed history with backpressure. |
| packages/durabletask-js/src/index.ts | Exports public payload APIs. |
| packages/durabletask-js/src/client/client.ts | Integrates payload processing and auto-purge configuration. |
| packages/durabletask-js-azuremanaged/test/unit/payload-builders.spec.ts | Tests builder configuration preservation. |
| packages/durabletask-js-azuremanaged/src/worker-builder.ts | Adds worker payload configuration. |
| packages/durabletask-js-azuremanaged/src/client-builder.ts | Adds client payload configuration. |
| packages/durabletask-js-azuremanaged/README.md | Documents builder options. |
| packages/durabletask-js-azuremanaged/CHANGELOG.md | Records builder support. |
| packages/durabletask-js-azureblob/tsconfig.json | Configures package TypeScript sources. |
| packages/durabletask-js-azureblob/tsconfig.build.json | Configures compiled package output. |
| packages/durabletask-js-azureblob/test/dotnet-payload-fixtures.json | Adds .NET gzip interoperability vectors. |
| packages/durabletask-js-azureblob/test/blob-payload-store.spec.ts | Tests storage, authentication, and deletion. |
| packages/durabletask-js-azureblob/test/azurite.spec.ts | Adds optional local storage integration tests. |
| packages/durabletask-js-azureblob/src/index.ts | Exports the Blob store. |
| packages/durabletask-js-azureblob/src/blob-payload-store.ts | Implements Blob storage and conditional cleanup. |
| packages/durabletask-js-azureblob/README.md | Documents configuration and cleanup contracts. |
| packages/durabletask-js-azureblob/package.json | Defines package dependencies and publishing metadata. |
| packages/durabletask-js-azureblob/LICENSE | Adds the package license. |
| packages/durabletask-js-azureblob/jest.config.js | Configures package tests. |
| packages/durabletask-js-azureblob/CHANGELOG.md | Records initial functionality. |
| package-lock.json | Registers the new workspace package. |
| jest.config.js | Adds Azure Blob test mapping. |
| internal/protocol/protos/large_payload_purge.proto | Defines the DTS cleanup protocol. |
| examples/azure-managed/README.md | Lists the payload sample. |
| examples/azure-managed/large-payloads/sample.json | Declares sample infrastructure requirements. |
| examples/azure-managed/large-payloads/README.md | Documents sample setup and cleanup. |
| examples/azure-managed/large-payloads/index.ts | Demonstrates a large-payload round trip. |
| eng/templates/build.yml | Packages and stages the extension. |
| eng/ci/release.yml | Adds extension publishing support. |
| CHANGELOG.md | Records payload and auto-purge features. |
| .github/workflows/prepare-release.yaml | Adds independent extension release preparation. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "@azure/storage-blob": "^12.26.0" | ||
| }, | ||
| "peerDependencies": { | ||
| "@microsoft/durabletask-js": ">=0.4.0" |
Comment on lines
+58
to
+60
| for (const event of action.getRewindorchestration()?.getNewhistoryList() ?? []) { | ||
| this._historyFields(event, fields); | ||
| } |
Reserve core 0.4.1 for the new payload APIs, raise both affected extension peer floors, and keep the Functions provider's exact core pin synchronized. Add packed-consumer npm checks rejecting published core 0.4.0 and loading the new core with both extensions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81e20512-d3a2-4f8e-a574-f396140ec9c7
Expose structural cause options instead of the ambient ES2022 ErrorOptions type. Extend the packed-consumer check to compile public declarations with ES2020 and skipLibCheck=false, verify the ES2022 global is absent, and retain runtime error-cause behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81e20512-d3a2-4f8e-a574-f396140ec9c7
This was referenced Oct 16, 2026
Member
Author
|
Superseded by six focused, dependency-stacked PRs:
Each PR targets its predecessor, so its diff is phase-specific rather than cumulative against main. Review/merge in order 1 -> 6. After a prerequisite merges, synchronize the next branch with main and retarget its PR before merging it; do not merge a downstream phase into its temporary predecessor branch. The complete feature is preserved, with the management-only purge-kind correction in phase 4 and unrelated formatting noise removed. Aggregate local checks and per-phase evidence are documented in the replacements. The final tip Closing this oversized PR in favor of the reviewable stack. The original |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What changed?
@microsoft/durabletask-js-azureblobpackage. Core users do not acquire Azure Storage dependencies.0.4.1for the new payload APIs. Both affected extensions require core>=0.4.1, and the Functions provider's exact core pin is synchronized. Real packed-consumer checks reject the incompatible published0.4.0and accept/load the new core and extensions.{ cause?: unknown }), so ES2020 declaration consumers do not need the global ES2022ErrorOptionstype.Why is this change needed?
microsoft/durabletask-dotnetat91739687ff647699079cdadd627daa00562982b7, including its interceptor, storage, and auto-purge tests.durabletask-payloadscontainer,managed_by=dts, credential-free v2 URLs, and legacy v1 reads. Cross-account access requires identity; configured account keys/SAS are not forwarded to another endpoint.Issues / work items
Project checklist
CHANGELOG.md>=0.4.1because its public payload option types do not exist in published0.4.0. The new Blob extension has the same minimum. Existing core APIs and disabled runtime behavior are preserved.0.4.1before the dependent packages; the Functions provider exact-pins0.4.1. Configure storage on every participating client/worker before writing externalized payloads, and enable auto-purge separately.AI-assisted code disclosure (required)
Was an AI tool used? (select one)
If AI was used:
AI verification (required if AI was used):
The human attestations above are intentionally left for the maintainer. Automated evidence is recorded below.
Testing
Automated tests
npm ci --offline --no-audit --no-fund— passed against the minimal lockfile.npm run test:unit— passed: 3,001 tests across 131 suites. The 10 opt-in Azurite tests were skipped in this command and run separately below.npm --registry=https://packagefeedproxy.microsoft.io/npm run test:payload-packages -- $oldCoreArchive($oldCoreArchiveis the actual npm-publishedmicrosoft-durabletask-js-0.4.0.tgz) — passed. npm rejects both extensions with old core usingERESOLVE; the new packed core/extensions install and load; public declarations compile withlib: ES2020, strict checking,skipLibCheck: false, and no ambient ES2022ErrorOptions; runtime error causes remain intact. Before the fixes, the check reproduced both acceptance of old core and TS2304 from the emitted declaration.npm --prefix test\e2e-functions\test-app run build— passed without changing the test-app's compiler target or libraries.npm run test:unit -w @microsoft/durabletask-js -- --runTestsByPath test\payloads.spec.ts test\client-error-cause.spec.ts test\orchestration-id-reuse-policy.spec.ts— passed: 84 tests after the declaration repair.npm run test:unit -w @microsoft/durabletask-js-azureblob— passed: 39 tests after the declaration repair; the 10 service-dependent tests remain opt-in.npm run test:azurite -w @microsoft/durabletask-js-azureblobwith a local-onlyDURABLETASK_BLOB_TEST_CONNECTION_STRING— passed: all 10 tests, including actual .NET-produced gzip blobs and ownership/snapshot cleanup.npm run build— passed for all five workspaces.npm run lint— passed.node_modules\.bin\prettier.cmd --check $files($files= staged TS/JS/JSON/Markdown files, excluding generated protobuf bindings) — passed.git diff --check origin/main...HEAD— passed.bash.exe -n tools\generate-grpc-javascript.sh— passed.npm pack --dry-run --json --workspace @microsoft/durabletask-js --workspace @microsoft/durabletask-js-azureblob— passed; core includes the new protocol, extension includes its compiled entry points, and test/reference artifacts are excluded.js-yaml; the new example type-checked with TypeScript; generated JS/.NET purge protocol descriptors compared byte-for-byte and matched (SHA-2565DDA0563E9A4F61340C6A975AF8CB5994863766674F24C0B561AB037C3335569).Manual validation (only if runtime/behavior changed)
Notes for reviewers
c9b53fc; live mixed-.NET/JS large-payload execution remains unverified.0.4.1is the first payload-capable source version and was verified not already published/tagged. Prepare that first core release with explicitversion: 0.4.1, then publish the extensions/Functions provider. Only core's own version advances here; extension version numbers stay independent. No npm package was published by this PR.d759df4exposed an ES2022ErrorOptionsleak in the public declaration. The structural signature repair is now covered by actual packed ES2020 consumers and the unchanged test-app build. All 30 executed hosted checks passed on final headc9b53fc(one inapplicable sample lane skipped); human review attestations remain unchecked.