Repository navigation
[Azure Functions] Reduce non-Latin payload inflation in df_dumps #291
Description
Parent: #269 — ask 4.
Problem
The reporter's activity results contained 2.55 MB of UTF-8 data, mostly Cyrillic, while the rejected orchestration completion request was 11.05 MB. The report identifies azure.functions._durable_functions.df_dumps using json.dumps with its default ensure_ascii=True as a source of inflation: a Cyrillic character that occupies two bytes in UTF-8 becomes a six-byte \uXXXX escape.
These figures describe the reported workload; the full request also has other content and overhead, so they should not be treated as a universal serialization ratio.
Agreed approach: explicit opt-in, unchanged defaults
The encoding API belongs in the shared azure-functions SDK, not in a competing Durable-specific serializer. Add a keyword-only option, df_dumps(value, *, ensure_ascii=True), covering every existing codec branch. The existing default remains byte-for-byte compatible. Explicit ensure_ascii=False emits ordinary Unicode literally while preserving transport-safe surrogate-code-unit escapes and the legacy ASCII DEL escape.
Wire this through a companion azure-functions-durable change using root-app configuration at startup, for example app.configure_serialization(ensure_ascii=False). The process-wide option must be applied consistently to synchronous/asynchronous clients, orchestration and entity workers, activity binding converters, and registered blueprints. Configure every worker consistently before invocations; conflicting settings must fail explicitly rather than silently changing encoding.
Applications that do not opt in retain current behavior, including with older azure-functions releases. Enabling the option requires an SDK exposing the upstream API and should fail clearly at startup if it is unavailable. Coordinate the eventual dependency minimum in a dedicated release change after the shared SDK capability is released; do not guess an unreleased version in the feature PR.
Upstream tracking: Azure/azure-functions-python-library#371.
Upstream implementation: Azure/azure-functions-python-library#372, now implementing the opt-in design.
Companion implementation: draft #292. It remains in draft until the upstream PR is merged and the shared SDK publishes a release containing the optional encoding API.
Compatibility
Existing ASCII-escaped and literal Unicode JSON deserialize to the same values. Local public-provider experiments exercised activity/sub-orchestration replay across encoding upgrades and rollbacks. The Functions/DTS compatibility assessment does not establish live production validation.
Because the codec belongs to a shared library, evaluation must also cover other consumers of its serialization API. Keeping the default unchanged avoids imposing a new byte representation on consumers that have not opted in. Applications deriving hashes, signatures, IDs, or replay decisions from serialized JSON still need upgrade/rollback guidance when enabling the option.
Acceptance criteria
- Preserve the shared codec's default and explicit
ensure_ascii=Trueoutput, envelopes, strict/loose rules, whitespace, and separators. - Add opt-in round-trip coverage for ASCII, Cyrillic, other scripts, Unicode keys, combining characters, and valid characters outside the basic multilingual plane.
- Keep surrogate code units and ASCII DEL escaped on the opt-in path so payloads remain transport-safe.
- Verify existing escaped payloads remain readable and assess persisted/replayed payload compatibility.
- Expose one consistent, startup-configured provider option across clients, workers, activities, entities, and blueprints, including large-payload externalization paths.
- Preserve compatibility with older shared-SDK releases when no opt-in is requested; report unsupported opt-in configuration explicitly.
- Demonstrate serialized UTF-8 size reduction without attributing unrelated request overhead to escaping.
- Document configuration scope, byte-dependent compatibility concerns, upstream release coordination, and the need for real-host staging validation.
See #269 for the full report. This is independent of large-payload externalization: reduced escaping does not eliminate aggregate request limits.
Activity
berndverst commentedon Oct 12, 2026
The encoding change belongs in the shared
azure-functionsSDK, notazure-functions-durable.df_dumpsis defined inazure.functions._durable_functions; the Durable provider delegates serialization to that codec. Upstream work is tracked in Azure/azure-functions-python-library#371.Our Functions/DTS compatibility assessment found no requirement for ASCII-only JSON for ordinary Unicode text. Existing escaped payloads remain readable, and local activity/sub-orchestration replay experiments worked across encoding upgrades and rollbacks. We therefore found no evidence that changing escaping alone would generally cause in-flight data loss. This is not a claim of live production validation.
Because the codec lives in a shared library, compatibility must also be evaluated for other consumers of that library, particularly any consumers of
df_dumpsthat rely on its current serialized representation. Durable-specific findings alone are not sufficient to approve a library-wide default change.A bare
ensure_ascii=Falsechange also needs safeguards: surrogate code units must remain escaped to avoid UTF-8/protobuf encoding failures. Serialized bytes change for non-ASCII text, so applications deriving hashes, signatures, IDs, or replay decisions from serialized JSON need upgrade guidance.A representative Cyrillic payload became approximately 67% smaller in serialized UTF-8 bytes. This reduces escaping overhead but does not eliminate aggregate request limits or replace large-payload externalization.