Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@

### Fixes

- Preserve received structured failure `properties` in task errors, retry callbacks, client
state/history, entity failures, and testing. Forward them with rethrown task failure chains.
Empty wire maps remain `undefined`; arbitrary JavaScript `Error` fields are not collected.
- Preserve nested `innerFailure` details in task errors, retry handlers and predicates,
client state/history, and in-memory testing. Uncaught or rethrown task failures retain
their received chain beneath the existing `TaskFailedError` wrapper. Retry decisions
Expand Down
23 changes: 20 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ Retries do not guarantee connection recovery, acceptance of expired tokens, or e
### Inspecting nested failures

Task errors expose an optional, read-only `innerFailure` chain, with `errorType`, `message`,
and optional `stackTrace` at each level. For an activity failure
and optional `stackTrace` and `properties` at each level. For an activity failure
`OrderFailed -> PaymentFailed -> ConnectionTimeout`, orchestrator code can inspect:

```typescript
Expand All @@ -202,7 +202,12 @@ try {
if (error instanceof TaskFailedError) {
const paymentFailure = error.details.innerFailure;
const rootFailure = paymentFailure?.innerFailure;
ctx.setCustomStatus({ rootErrorType: rootFailure?.errorType });
// If the backend supplied a structured "code" property, narrow its unknown value.
const code = paymentFailure?.properties?.["code"];
ctx.setCustomStatus({
rootErrorType: rootFailure?.errorType,
paymentCode: typeof code === "string" ? code : undefined,
});
}
throw error;
}
Expand All @@ -216,13 +221,25 @@ task details under `state.failureDetails.innerFailure`.

Missing inner failures remain `undefined`; existing two- and three-argument
`FailureDetails` constructors still work, with an optional fourth argument for the inner
details. Remote failures are not reconstructed as JavaScript `Error.cause` objects.
details and fifth argument for properties. Remote failures are not reconstructed as
JavaScript `Error.cause` objects.
The existing writer still limits ordinary `Error.cause` chains to ten inner levels;
reading or forwarding already-received failure details does not add a truncation limit.
If user code mutates received details into a cycle, forwarding preserves each unique
failure and ends the chain with `errorType: "CircularFailureDetails"` and
`message: "A circular innerFailure reference was detected."` rather than looping.

`properties` is a `Readonly<Record<string, unknown>>` containing values already received
from the backend: strings, numbers, booleans, `null`, arrays, and objects. Empty or absent
wire maps become `undefined`. Wire strings stay strings, including .NET `dt:`/`dto:`
date encodings; this is not CLR type reconstruction or a custom data converter.
The same properties reach retry callbacks, client state/history, entity failure details,
and test adapters. Ordinary JavaScript `Error` fields are **not** automatically collected.
Forwarding user-mutated properties rejects unsupported values (such as `Date`, `bigint`,
or `undefined`) and circular objects instead of silently dropping data. The current
protobuf dependency cannot retain the reserved `__proto__` map key on receipt; adding
that key before forwarding also raises a serialization error.

### Reusing orchestration instance IDs

Set the top-level `dedupeStatuses` start option when an instance ID may be reused. The list
Expand Down
2 changes: 2 additions & 0 deletions packages/azure-functions-durable/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

### Fixes

- Preserve received failure `properties` in `durable-functions/testing` results, including
nested failures. This does not collect arbitrary JavaScript `Error` fields.
- Forward the optional `reason` from the classic `suspend()` / `resume()` aliases to the
core client instead of ignoring it, preserving literal strings including empty strings.
- Preserve nested task `innerFailure` details in `durable-functions/testing` results.
Expand Down
1 change: 1 addition & 0 deletions packages/azure-functions-durable/src/testing/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ export async function runOrchestrator<TOutput = unknown, TInput = unknown>(
message: state.failureDetails.message,
stackTrace: state.failureDetails.stackTrace,
innerFailure: state.failureDetails.innerFailure,
properties: state.failureDetails.properties,
}
: undefined,
};
Expand Down
37 changes: 37 additions & 0 deletions packages/azure-functions-durable/test/unit/testing.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
import { InvocationContext } from "@azure/functions";
import {
InMemoryOrchestrationBackend,
FailureDetails,
OrchestrationState,
OrchestrationStatus,
TestOrchestrationClient,
TestOrchestrationWorker,
TOrchestrator,
Expand Down Expand Up @@ -53,6 +56,40 @@ describe("durable-functions/testing", () => {
});

describe("runOrchestrator", () => {
it("preserves received root and nested properties in its failure projection", async () => {
const inner = new FailureDetails("inner", "InnerError", undefined, undefined, { retryable: false });
const failure = new FailureDetails("remote", "RemoteError", undefined, inner, { code: "RemoteCode" });
const state = new OrchestrationState(
"remote",
"orchestrator",
OrchestrationStatus.FAILED,
new Date(0),
new Date(0),
undefined,
undefined,
undefined,
failure,
);
// Inject received client state to isolate the adapter's public projection.
const wait = jest
.spyOn(TestOrchestrationClient.prototype, "waitForOrchestrationCompletion")
.mockResolvedValue(state);
try {
const orchestrator: TOrchestrator = async function* (ctx) {
yield ctx.createTimer(0);
};
const result = await runOrchestrator(orchestrator);
expect(result.failure).toMatchObject({
errorType: "RemoteError",
message: "remote",
properties: { code: "RemoteCode" },
innerFailure: { errorType: "InnerError", message: "inner", properties: { retryable: false } },
});
} finally {
wait.mockRestore();
}
});

it("inherits the core three-day timer default, like the Functions worker", async () => {
const complete = jest.spyOn(InMemoryOrchestrationBackend.prototype, "completeOrchestration");
const day = 24 * 60 * 60 * 1000;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import { EntityInstanceId } from "./entity-instance-id";
import * as pb from "../proto/orchestrator_service_pb";
import { convertFailureProperties } from "../utils/failure-details.util";

/**
* Details about a task failure.
Expand Down Expand Up @@ -31,6 +32,8 @@ export interface TaskFailureDetails {
* Details about an inner failure, if any.
*/
readonly innerFailure?: TaskFailureDetails;
/** Structured backend values, if any; empty wire maps are undefined. Not collected from Error fields. */
readonly properties?: Readonly<Record<string, unknown>>;
}

/**
Expand All @@ -49,6 +52,7 @@ export function createTaskFailureDetails(proto: pb.TaskFailureDetails | undefine
errorMessage: proto.getErrormessage(),
stackTrace: proto.getStacktrace()?.getValue(),
innerFailure: createTaskFailureDetails(proto.getInnerfailure()),
properties: convertFailureProperties(proto),
};
}

Expand Down
20 changes: 19 additions & 1 deletion packages/durabletask-js/src/task/failure-details.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,33 @@ export interface TaskFailureDetails {
readonly stackTrace?: string;
/** Details of the underlying failure, if supplied by the task or backend. */
readonly innerFailure?: TaskFailureDetails;
/**
* Structured values received from the backend, not collected from JavaScript Error fields.
* Empty wire maps are undefined. Values are protobuf Value data (strings, numbers, booleans,
* null, arrays, or objects); narrow unknown values before using them.
*/
readonly properties?: Readonly<Record<string, unknown>>;
}

export class FailureDetails implements TaskFailureDetails {
private _message: string;
private _errorType: string;
private _stackTrace: string | undefined;
private _innerFailure: TaskFailureDetails | undefined;
private _properties: Readonly<Record<string, unknown>> | undefined;

constructor(message: string, errorType: string, stackTrace?: string, innerFailure?: TaskFailureDetails) {
constructor(
message: string,
errorType: string,
stackTrace?: string,
innerFailure?: TaskFailureDetails,
properties?: Readonly<Record<string, unknown>>,
) {
this._message = message;
this._errorType = errorType;
this._stackTrace = stackTrace;
this._innerFailure = innerFailure;
this._properties = properties;
}

get message(): string {
Expand All @@ -45,4 +59,8 @@ export class FailureDetails implements TaskFailureDetails {
get innerFailure(): TaskFailureDetails | undefined {
return this._innerFailure;
}

get properties(): Readonly<Record<string, unknown>> | undefined {
return this._properties;
}
}
1 change: 1 addition & 0 deletions packages/durabletask-js/src/task/retry-handler-task.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ export class RetryHandlerTask<T> extends RetryTaskBase<T> {
message: details.message,
stackTrace: details.stackTrace,
innerFailure: details.innerFailure,
properties: details.properties,
};

const totalRetryTimeMs = currentTime.getTime() - this.startTime.getTime();
Expand Down
1 change: 1 addition & 0 deletions packages/durabletask-js/src/task/retryable-task.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ export class RetryableTask<T> extends RetryTaskBase<T> {
message: details.message,
stackTrace: details.stackTrace,
innerFailure: details.innerFailure,
properties: details.properties,
};

if (!this._retryPolicy.shouldRetry(failureDetails)) {
Expand Down
46 changes: 46 additions & 0 deletions packages/durabletask-js/src/utils/failure-details.util.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import { JavaScriptValue, Value } from "google-protobuf/google/protobuf/struct_pb";
import * as pb from "../proto/orchestrator_service_pb";
import { FailureDetails } from "../task/failure-details";

Expand All @@ -16,5 +17,50 @@ export function convertFailureDetails(details: pb.TaskFailureDetails | undefined
details.getErrortype(),
details.getStacktrace()?.getValue(),
convertFailureDetails(details.getInnerfailure()),
convertFailureProperties(details),
);
}

export function convertFailureProperties(
details: pb.TaskFailureDetails,
): Readonly<Record<string, unknown>> | undefined {
const properties = details.getPropertiesMap();
return properties.getLength() === 0
? undefined
: Object.fromEntries(Array.from(properties.entries(), ([key, value]) => [key, value.toJavaScript()]));
}

export function setFailureProperties(
details: pb.TaskFailureDetails,
properties: Readonly<Record<string, unknown>> | undefined,
): void {
if (properties === undefined) return;
for (const [key, value] of Object.entries(properties)) {
validatePropertyValue(value, key);
details.getPropertiesMap().set(key, Value.fromJavaScript(value));
}
}

function validatePropertyValue(
value: unknown,
key: string,
ancestors = new Set<object>(),
): asserts value is JavaScriptValue {
// google-protobuf maps silently drop this key instead of preserving it as data.
if (key === "__proto__") throw new TypeError("google-protobuf cannot serialize a '__proto__' failure property.");
if (value === null || typeof value === "string" || typeof value === "number" || typeof value === "boolean") return;
if (
typeof value !== "object" ||
(!Array.isArray(value) &&
Object.getPrototypeOf(value) !== Object.prototype &&
Object.getPrototypeOf(value) !== null)
) {
throw new TypeError("Failure properties must contain only protobuf Value-compatible data.");
}
if (ancestors.has(value)) throw new TypeError("Cannot serialize circular failure properties.");
ancestors.add(value);
for (const [childKey, child] of Object.entries(value)) {
validatePropertyValue(child, childKey, ancestors);
}
ancestors.delete(value);
}
2 changes: 2 additions & 0 deletions packages/durabletask-js/src/utils/pb-helper.util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { ActivityNotRegisteredError } from "../worker/exception/activity-not-reg
import { OrchestratorNotRegisteredError } from "../worker/exception/orchestrator-not-registered-error";
import { TaskFailedError } from "../task/exception/task-failed-error";
import { TaskFailureDetails } from "../task/failure-details";
import { setFailureProperties } from "./failure-details.util";

export function newOrchestratorStartedEvent(timestamp?: Date | null): pb.HistoryEvent {
const ts = new Timestamp();
Expand Down Expand Up @@ -258,6 +259,7 @@ function buildFailureDetails(e: unknown, depth: number): pb.TaskFailureDetails {
if (details.stackTrace !== undefined) {
inner.setStacktrace(new StringValue().setValue(details.stackTrace));
}
setFailureProperties(inner, details.properties);
current.setInnerfailure(inner);
current = inner;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
import { TaskFailedError } from "../src/task/exception/task-failed-error";
import * as pb from "../src/proto/orchestrator_service_pb";
import { StringValue } from "google-protobuf/google/protobuf/wrappers_pb";
import { Value } from "google-protobuf/google/protobuf/struct_pb";

describe("EntityOperationFailedException", () => {
describe("constructor", () => {
Expand Down Expand Up @@ -155,11 +156,13 @@ describe("createTaskFailureDetails", () => {
const innerProto = new pb.TaskFailureDetails();
innerProto.setErrortype("InnerError");
innerProto.setErrormessage("Inner message");
innerProto.getPropertiesMap().set("retryable", Value.fromJavaScript(false));

const proto = new pb.TaskFailureDetails();
proto.setErrortype("OuterError");
proto.setErrormessage("Outer message");
proto.setInnerfailure(innerProto);
proto.getPropertiesMap().set("code", Value.fromJavaScript("Conflict"));

// Act
const result = createTaskFailureDetails(proto);
Expand All @@ -169,6 +172,8 @@ describe("createTaskFailureDetails", () => {
expect(result!.innerFailure).toBeDefined();
expect(result!.innerFailure!.errorType).toBe("InnerError");
expect(result!.innerFailure!.errorMessage).toBe("Inner message");
expect(Reflect.get(result!, "properties")).toEqual({ code: "Conflict" });
expect(Reflect.get(result!.innerFailure!, "properties")).toEqual({ retryable: false });
});

it("should handle missing optional fields", () => {
Expand All @@ -185,5 +190,6 @@ describe("createTaskFailureDetails", () => {
expect(result).toBeDefined();
expect(result!.stackTrace).toBeUndefined();
expect(result!.innerFailure).toBeUndefined();
expect(Reflect.get(result!, "properties")).toBeUndefined();
});
});
Loading
Loading