Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .github/workflows/prepare-release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# 1. Bumps the selected package's version in its own package.json
# 2. Generates a changelog from git history since the package's last release tag
# 3. Updates the package's changelog file and creates a release branch + package-scoped tag
# Packages are versioned and tagged independently (tag prefixes: v, azuremanaged-v, durable-functions-v).
# Packages are versioned and tagged independently (tag prefixes: v, azureblob-v, azuremanaged-v, durable-functions-v).
# After the workflow completes, manually open a PR from the release branch to main, then publish
# from the package directory using the npm command shown in the run summary (mind the dist-tag).

Expand All @@ -19,6 +19,7 @@ on:
type: choice
options:
- durabletask-js
- durabletask-js-azureblob
- durabletask-js-azuremanaged
- azure-functions-durable
version:
Expand Down Expand Up @@ -60,14 +61,20 @@ jobs:
TAG_PREFIX="azuremanaged-v"
CHANGELOG_PATH="packages/durabletask-js-azuremanaged/CHANGELOG.md"
;;
durabletask-js-azureblob)
PKG_DIR="packages/durabletask-js-azureblob"
NPM_NAME="@microsoft/durabletask-js-azureblob"
TAG_PREFIX="azureblob-v"
CHANGELOG_PATH="packages/durabletask-js-azureblob/CHANGELOG.md"
;;
azure-functions-durable)
PKG_DIR="packages/azure-functions-durable"
NPM_NAME="durable-functions"
TAG_PREFIX="durable-functions-v"
CHANGELOG_PATH="packages/azure-functions-durable/CHANGELOG.md"
;;
*)
echo "::error::Unknown package '$PACKAGE'. Expected one of: durabletask-js, durabletask-js-azuremanaged, azure-functions-durable."
echo "::error::Unknown package '$PACKAGE'. Expected one of: durabletask-js, durabletask-js-azureblob, durabletask-js-azuremanaged, azure-functions-durable."
exit 1
;;
esac
Expand Down
22 changes: 10 additions & 12 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,12 @@

### New

- Add reserved cleanup worker tasks with bounded deletion chunks, retry/quarantine handling,
configuration-preserving continue-as-new, authenticated current-channel RPCs and filter/version isolation.
This completes the cleanup runtime; aggregate release/consumer validation follows in phase 6.
- Add the DTS payload-purge protocol and `setLargePayloadAutoPurge()` client lifecycle,
including setting-first enable/disable and reserved-instance identity verification.
A compatible .NET worker or the later cleanup-worker phase is required; do not publish partial phases.
- Add payload storage contracts and bounded protobuf payload processing, with inclusive UTF-8
thresholds, explicit storage/size failures, and non-retriable worker upload-failure conversion.
This is phase 1 of the external-payload feature; transport, providers, and cleanup follow separately.
Reserve core `0.4.1` for the new APIs; do not publish partial feature phases.
- Add opt-in `payloads` storage to gRPC clients/workers, with transparent UTF-8-size-based
externalization and awaited query/history/work-item hydration across orchestration, activity,
and entity V1/V2 payloads. Add the optional `@microsoft/durabletask-js-azureblob` storage package.
- Add .NET-aligned `setLargePayloadAutoPurge()`, reserved replay-safe cleanup tasks, and DTS
lifecycle transport. Storage configuration alone does not enable deletion; cleanup checks
blob ownership and ETags, preserves uncertain outcomes, and leaves legacy v1 blobs untouched.
- Expose readonly `OrchestrationContext.sourceInstanceId` from the existing protobuf
request properties on initial execution and replay, independently of parent identity.
Strings are preserved verbatim; null or missing metadata returns `undefined`.
Expand Down Expand Up @@ -53,8 +49,10 @@

### Fixes

- Determine single-instance purge routing from the instance ID regardless of payload configuration,
matching .NET management clients: ordinary IDs are orchestrations and `@name@key` IDs are entities.
- Determine single-instance purge routing from ordinary/entity IDs regardless of payload storage,
matching .NET management clients without changing purge timeout/completion semantics.
- Keep `PayloadStorageException` cause options structural so its public declarations remain
usable by ES2020 consumers without the global ES2022 `ErrorOptions` type.
- Apply payload externalization and UTF-8 size limits to native continue-as-new carryover history.
Event metadata, ordering, and hydrated payload values are preserved.
- Validate every eligible payload's UTF-8 size before starting uploads, preventing storage writes
Expand Down
47 changes: 16 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,37 +23,22 @@ const entityResponseBytes = await worker.processEntityBatchRequest(entityBatchRe

`TaskHubGrpcClient` already exposes orchestration start/query/event/terminate/suspend/resume/purge APIs and entity signal/read/query/clean APIs through its existing `hostAddress` and `metadataGenerator` options. Host integrations that need task-hub routing metadata should provide it through `metadataGenerator`, keeping host-specific metadata policy outside the core client. Azure-managed scheduler connection strings remain in `@microsoft/durabletask-js-azuremanaged`.

## External payload transport

Core clients and workers accept opt-in `payloads: { store }` options, where `store` implements
the exported `PayloadStore` contract. The same store must be configured on every participant.
The default threshold is 256 KiB of serialized UTF-8 bytes (inclusive), with a 10 MiB maximum.
Requests are externalized before sending; queries, history chunks and work items are hydrated
before conversion or execution. Empty payloads remain inline, and disabling the option preserves
the existing transport behavior.

Azure-managed builders expose `.useExternalizedPayloads({ store })` and require core `>=0.4.1`
for the public payload types. A custom/in-memory store or the optional Azure Blob provider can
implement storage independently of the transport. Complete the six-phase feature stack before release.

### Payload cleanup client protocol

`client.setLargePayloadAutoPurge(enabled, batchSize?, signal?)` manages the authenticated DTS
task-hub cleanup setting. Enabling writes the setting first, starts/deduplicates and verifies
the reserved Running orchestration, then signals its batch size. Disabling only writes the
backend setting. These steps are not transactional; earlier successful steps are not rolled back.

This client operation does not require blob credentials. It requires a backend implementing
the DTS `LargePayloadPurge` service and a compatible worker. Payload-enabled JavaScript workers
now register the reserved cleanup orchestrator and activities automatically, preserve their
wildcard filters and bypass business version policy without changing customer filters.
Deletion verifies ownership and ETags, batches 50 tokens per activity and limits concurrency;
the runner backs off empty/retry-only batches and preserves configuration through continue-as-new.
Do not publish the feature until all six phases are complete.

Single-instance purges now identify the instance kind even on management-only clients without
payload storage: ordinary IDs set `isOrchestration=true`, while `@name@key` entity IDs set it false.
The existing timeout and completion-status behavior is unchanged.
## Large payloads

Clients and workers can opt into transparent external payload storage with the `payloads`
option. The optional [`@microsoft/durabletask-js-azureblob`](./packages/durabletask-js-azureblob)
package implements .NET-compatible Azure Blob tokens and UTF-8/gzip storage. The default
externalization threshold is 256 KiB (inclusive); the maximum payload size is 10 MiB.
Core users do not acquire Azure Storage dependencies; payload transport remains opt-in.

Configure all participating clients and workers before storing externalized payloads.
Azure-managed builders expose `.useExternalizedPayloads({ store })`. Cleanup is separately
opt-in through `client.setLargePayloadAutoPurge(true)` and requires a supporting DTS backend.
Single-instance purges now route ordinary/entity IDs correctly even without payload storage.
See the package's [configuration, interoperability, and cleanup contract](./packages/durabletask-js-azureblob/README.md).

This feature is reviewed as a six-phase dependency stack. Merge all phases before preparing
a release; publish the completed core `0.4.1` before the dependent packages.

## Long durable timers

Expand Down
Loading
Loading