Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@

### Fixes

- Preserve nested `innerFailure` details in task errors, retry handlers and predicates,
client state/history, and in-memory testing. Uncaught or rethrown task failures retain
their received chain beneath the existing `TaskFailedError` wrapper. Retry decisions
and the ordinary `Error.cause` serialization depth limit are unchanged.
User-mutated circular detail chains terminate with an explicit `CircularFailureDetails`
marker after the unique prefix, without imposing a depth limit on finite received chains.
- Preserve explicit empty activity and child versions on the protobuf wire, including the
unversioned child default, so concrete empty-version work-item filters can match them.
- Reject uninitialized `whenAll` results after a canceled child's completion callback throws,
Expand Down
36 changes: 36 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,42 @@ User code and metadata generation are not canceled; if metadata finishes after s
the response RPC is not started. Channel retirement and backend lock durations are unchanged.
Retries do not guarantee connection recovery, acceptance of expired tokens, or exactly-once execution.

### Inspecting nested failures

Task errors expose an optional, read-only `innerFailure` chain, with `errorType`, `message`,
and optional `stackTrace` at each level. For an activity failure
`OrderFailed -> PaymentFailed -> ConnectionTimeout`, orchestrator code can inspect:

```typescript
import { TaskFailedError } from "@microsoft/durabletask-js";

try {
yield ctx.callActivity("placeOrder");
} catch (error) {
if (error instanceof TaskFailedError) {
const paymentFailure = error.details.innerFailure;
const rootFailure = paymentFailure?.innerFailure;
ctx.setCustomStatus({ rootErrorType: rootFailure?.errorType });
}
throw error;
}
```

The same chain is available as `context.lastFailure.innerFailure` in retry handlers and
`failure.innerFailure` in `RetryPolicy.handleFailure`. Inspecting nested failures does not
change retry decisions automatically. Client state and history also preserve the chain;
an uncaught or rethrown task error retains its `TaskFailedError` wrapper, with the original
task details under `state.failureDetails.innerFailure`.

Missing inner failures remain `undefined`; existing two- and three-argument
`FailureDetails` constructors still work, with an optional fourth argument for the inner
details. Remote failures are not reconstructed as JavaScript `Error.cause` objects.
The existing writer still limits ordinary `Error.cause` chains to ten inner levels;
reading or forwarding already-received failure details does not add a truncation limit.
If user code mutates received details into a cycle, forwarding preserves each unique
failure and ends the chain with `errorType: "CircularFailureDetails"` and
`message: "A circular innerFailure reference was detected."` rather than looping.

### Reusing orchestration instance IDs

Set the top-level `dedupeStatuses` start option when an instance ID may be reused. The list
Expand Down
1 change: 1 addition & 0 deletions packages/azure-functions-durable/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

### Fixes

- Preserve nested task `innerFailure` details in `durable-functions/testing` results.
- Inherit core version-aware replay dispatch and worker child defaults in the embedded
`DurableFunctionsWorker`, including classic-context wrappers. Host `app.*` function registrations
remain name-only; see README for the boundary and activity-version migration guidance.
Expand Down
1 change: 1 addition & 0 deletions packages/azure-functions-durable/src/testing/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ export async function runOrchestrator<TOutput = unknown, TInput = unknown>(
errorType: state.failureDetails.errorType,
message: state.failureDetails.message,
stackTrace: state.failureDetails.stackTrace,
innerFailure: state.failureDetails.innerFailure,
}
: undefined,
};
Expand Down
7 changes: 6 additions & 1 deletion packages/azure-functions-durable/test/unit/testing.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ describe("durable-functions/testing", () => {
const result = await runOrchestrator(orchestrator, {
activities: {
fail: () => {
throw new TypeError("activity failed");
throw new TypeError("activity failed", { cause: new Error("connection timed out") });
},
},
});
Expand All @@ -151,6 +151,11 @@ describe("durable-functions/testing", () => {
expect(result.failure).toMatchObject({
errorType: expect.any(String),
message: expect.stringContaining("activity failed"),
innerFailure: {
errorType: "TypeError",
message: "activity failed",
innerFailure: { errorType: "Error", message: "connection timed out", innerFailure: undefined },
},
});
});

Expand Down
12 changes: 2 additions & 10 deletions packages/durabletask-js/src/client/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import { TerminateInstanceOptions, isTerminateInstanceOptions } from "../orchest
import { callWithMetadata, MetadataGenerator } from "../utils/grpc-helper.util";
import { OrchestrationQuery, ListInstanceIdsOptions, DEFAULT_PAGE_SIZE } from "../orchestration/orchestration-query";
import { Page, AsyncPageable, createAsyncPageable } from "../orchestration/page";
import { FailureDetails } from "../task/failure-details";
import { convertFailureDetails } from "../utils/failure-details.util";
import { HistoryEvent } from "../orchestration/history-event";
import { convertProtoHistoryEvent } from "../utils/history-event-converter";
import { Logger, ConsoleLogger } from "../types/logger.type";
Expand Down Expand Up @@ -1376,15 +1376,7 @@ export class TaskHubGrpcClient {
}

// Extract failure details if present
let failureDetails;
const protoFailureDetails = protoState.getFailuredetails();
if (protoFailureDetails) {
failureDetails = new FailureDetails(
protoFailureDetails.getErrormessage(),
protoFailureDetails.getErrortype(),
protoFailureDetails.getStacktrace()?.getValue(),
);
}
const failureDetails = convertFailureDetails(protoState.getFailuredetails());

const tags = mapToRecord(protoState.getTagsMap());

Expand Down
13 changes: 2 additions & 11 deletions packages/durabletask-js/src/orchestration/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// Licensed under the MIT License.

import * as pb from "../proto/orchestrator_service_pb";
import { FailureDetails } from "../task/failure-details";
import { convertFailureDetails } from "../utils/failure-details.util";
import { fromProtobuf } from "./enum/orchestration-status.enum";
import { OrchestrationState } from "./orchestration-state";
import { mapToRecord } from "../utils/tags.util";
Expand All @@ -16,16 +16,7 @@ export function newOrchestrationState(
}

const state = res.getOrchestrationstate();
let failureDetails;

const protoFailureDetails = state?.getFailuredetails();
if (protoFailureDetails) {
failureDetails = new FailureDetails(
protoFailureDetails.getErrormessage(),
protoFailureDetails.getErrortype(),
protoFailureDetails.getStacktrace()?.getValue(),
);
}
const failureDetails = convertFailureDetails(state?.getFailuredetails());

const tsCreated = state?.getCreatedtimestamp();
const tsUpdated = state?.getLastupdatedtimestamp();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import * as pb from "../../proto/orchestrator_service_pb";
import { FailureDetails } from "../failure-details";
import { convertFailureDetails } from "../../utils/failure-details.util";

export class TaskFailedError extends Error {
private _details: FailureDetails;
Expand All @@ -11,11 +12,7 @@ export class TaskFailedError extends Error {
super(message);
this.name = "TaskFailedError";

this._details = new FailureDetails(
details.getErrormessage(),
details.getErrortype(),
details?.getStacktrace()?.getValue(),
);
this._details = convertFailureDetails(details);
}

get details(): FailureDetails {
Expand Down
13 changes: 11 additions & 2 deletions packages/durabletask-js/src/task/failure-details.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

/**
* Interface representing task failure details.
* This is used for retry handlers to inspect failure information.
* Used by task errors and retry handlers to inspect failure information,
* including the optional chain of underlying failures.
*/
export interface TaskFailureDetails {
/** The type/class name of the error */
Expand All @@ -12,17 +13,21 @@ export interface TaskFailureDetails {
readonly message: string;
/** The stack trace, if available */
readonly stackTrace?: string;
/** Details of the underlying failure, if supplied by the task or backend. */
readonly innerFailure?: TaskFailureDetails;
}

export class FailureDetails implements TaskFailureDetails {
private _message: string;
private _errorType: string;
private _stackTrace: string | undefined;
private _innerFailure: TaskFailureDetails | undefined;

constructor(message: string, errorType: string, stackTrace?: string) {
constructor(message: string, errorType: string, stackTrace?: string, innerFailure?: TaskFailureDetails) {
this._message = message;
this._errorType = errorType;
this._stackTrace = stackTrace;
this._innerFailure = innerFailure;
}

get message(): string {
Expand All @@ -36,4 +41,8 @@ export class FailureDetails implements TaskFailureDetails {
get stackTrace(): string | undefined {
return this._stackTrace;
}

get innerFailure(): TaskFailureDetails | undefined {
return this._innerFailure;
}
}
9 changes: 6 additions & 3 deletions packages/durabletask-js/src/task/retry-handler-task.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { RetryTaskBase, RetryTaskType } from "./retry-task-base";
import { AsyncRetryHandler, RetryHandlerResult } from "./retry/retry-handler";
import { createRetryContext } from "./retry/retry-context";
import { TaskFailureDetails } from "./failure-details";
import { convertFailureDetails } from "../utils/failure-details.util";

/**
* A task that uses an AsyncRetryHandler for imperative retry control.
Expand Down Expand Up @@ -73,10 +74,12 @@ export class RetryHandlerTask<T> extends RetryTaskBase<T> {
return false;
}

const details = convertFailureDetails(this.lastFailure);
const failureDetails: TaskFailureDetails = {
errorType: this.lastFailure.getErrortype() || "Error",
message: this.lastFailure.getErrormessage() || "",
stackTrace: this.lastFailure.getStacktrace()?.getValue(),
errorType: details.errorType || "Error",
message: details.message,
stackTrace: details.stackTrace,
innerFailure: details.innerFailure,
};

const totalRetryTimeMs = currentTime.getTime() - this.startTime.getTime();
Expand Down
2 changes: 1 addition & 1 deletion packages/durabletask-js/src/task/retry/retry-context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ export interface RetryContext {

/**
* The details of the previous task failure.
* Contains the error type, message, and stack trace.
* Contains the error type, message, stack trace, and optional inner failure chain.
*/
readonly lastFailure: TaskFailureDetails;

Expand Down
9 changes: 6 additions & 3 deletions packages/durabletask-js/src/task/retryable-task.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import { RetryTaskBase, RetryTaskType } from "./retry-task-base";
import { RetryPolicy } from "./retry/retry-policy";
import * as pb from "../proto/orchestrator_service_pb";
import { convertFailureDetails } from "../utils/failure-details.util";

/**
* A task that can be retried according to a declarative retry policy.
Expand Down Expand Up @@ -66,10 +67,12 @@ export class RetryableTask<T> extends RetryTaskBase<T> {

// Check if handleFailure predicate says we should NOT retry this failure type
if (this.lastFailure) {
const details = convertFailureDetails(this.lastFailure);
const failureDetails = {
errorType: this.lastFailure.getErrortype() || "Error",
message: this.lastFailure.getErrormessage() || "",
stackTrace: this.lastFailure.getStacktrace()?.getValue(),
errorType: details.errorType || "Error",
message: details.message,
stackTrace: details.stackTrace,
innerFailure: details.innerFailure,
};

if (!this._retryPolicy.shouldRetry(failureDetails)) {
Expand Down
2 changes: 1 addition & 1 deletion packages/durabletask-js/src/testing/in-memory-backend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1245,7 +1245,7 @@ export class InMemoryOrchestrationBackend {
event = pbh.newSubOrchestrationCompletedEvent(taskId, subInstance.output);
} else {
const error = new Error(subInstance.failureDetails?.getErrormessage() || "Sub-orchestration failed");
event = pbh.newSubOrchestrationFailedEvent(taskId, error);
event = pbh.newSubOrchestrationFailedEvent(taskId, error, subInstance.failureDetails);
}
parentInstance.pendingEvents.push(event);
parentInstance.lastUpdatedAt = new Date();
Expand Down
11 changes: 2 additions & 9 deletions packages/durabletask-js/src/testing/test-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { getName } from "../task";
import { TOrchestrator } from "../types/orchestrator.type";
import { TInput } from "../types/input.type";
import { OrchestrationState } from "../orchestration/orchestration-state";
import { FailureDetails } from "../task/failure-details";
import { convertFailureDetails } from "../utils/failure-details.util";
import { EntityInstanceId } from "../entities/entity-instance-id";
import { EntityMetadata } from "../entities/entity-metadata";
import { InMemoryOrchestrationBackend, OrchestrationInstance } from "./in-memory-backend";
Expand Down Expand Up @@ -229,14 +229,7 @@ export class TestOrchestrationClient {
}

private toOrchestrationState(instance: OrchestrationInstance, fetchPayloads: boolean): OrchestrationState {
let failureDetails: FailureDetails | undefined;
if (instance.failureDetails) {
failureDetails = new FailureDetails(
instance.failureDetails.getErrormessage(),
instance.failureDetails.getErrortype(),
instance.failureDetails.getStacktrace()?.getValue(),
);
}
const failureDetails = convertFailureDetails(instance.failureDetails);

return new OrchestrationState(
instance.instanceId,
Expand Down
20 changes: 20 additions & 0 deletions packages/durabletask-js/src/utils/failure-details.util.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import * as pb from "../proto/orchestrator_service_pb";
import { FailureDetails } from "../task/failure-details";

export function convertFailureDetails(details: pb.TaskFailureDetails): FailureDetails;
export function convertFailureDetails(details: pb.TaskFailureDetails | undefined): FailureDetails | undefined;
export function convertFailureDetails(details: pb.TaskFailureDetails | undefined): FailureDetails | undefined {
if (!details) {
return undefined;
}

return new FailureDetails(
details.getErrormessage(),
details.getErrortype(),
details.getStacktrace()?.getValue(),
convertFailureDetails(details.getInnerfailure()),
);
}
12 changes: 1 addition & 11 deletions packages/durabletask-js/src/utils/history-event-converter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import {
ParentInstanceInfo,
TraceContext,
} from "../orchestration/history-event";
import { FailureDetails } from "../task/failure-details";
import { convertFailureDetails } from "./failure-details.util";

// Map OrchestrationStatus enum values to their string names
const ORCHESTRATION_STATUS_MAP: Record<number, string> = {
Expand Down Expand Up @@ -498,16 +498,6 @@ function convertTraceContext(traceContext: pb.TraceContext): TraceContext {
};
}

function convertFailureDetails(details: pb.TaskFailureDetails | undefined): FailureDetails | undefined {
if (!details) return undefined;

return new FailureDetails(
details.getErrormessage(),
details.getErrortype(),
details.getStacktrace()?.getValue(),
);
}

function convertTagsMap(tagsMap: ReturnType<pb.ExecutionStartedEvent["getTagsMap"]>): Record<string, string> | undefined {
const result: Record<string, string> = {};
let hasEntries = false;
Expand Down
Loading
Loading