Skip to content

Commit bb2a2bc

Browse files
andystaplesCopilot
andcommitted
fix: complete shared and duplicate when_all tasks
Notify all dependent composites in subscription order, count duplicate task references by identity, and detach completed races without disturbing other observers. Preserve callback failures and document replay migration guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent e1bb6bb commit bb2a2bc

9 files changed

Lines changed: 454 additions & 12 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,26 @@ ADDED
1313

1414
FIXED
1515

16+
- Fixed `when_all` hanging when pending tasks are shared between groups or
17+
appear multiple times in one group. Results preserve input order and duplicate
18+
positions without scheduling additional activities. Completed `when_any` groups
19+
stop observing losing children without disturbing other groups, and a throwing
20+
completion callback no longer prevents other observers from being notified.
21+
22+
> [!WARNING]
23+
> **Replay-breaking bug fix:** Shared nested races can select a different direct
24+
> child during replay. For example, with `first = when_any([timer, work])` and
25+
> `second = when_any([first, timer])`, timer completion now notifies `first`,
26+
> allowing `second` to select `first` rather than `timer`. Existing downstream
27+
> history can then fail replay with `NonDeterminismError`. Disjoint task groups
28+
> are unchanged.
29+
>
30+
> Drain affected instances on the previous SDK, or keep their task hub on the
31+
> previous SDK and use a separate task hub for new instances. Already-stuck
32+
> instances may need deliberate recovery. Pin or lock the core `durabletask`
33+
> dependency, including when using a provider. Provider dependency minimums
34+
> will be coordinated in a dedicated release PR after the core fix is released.
35+
1636
- Single-instance `purge_orchestration()` requests now explicitly target
1737
orchestrations rather than entities in both synchronous and asynchronous clients.
1838
- Timer callbacks no longer schedule additional long-timer chunks, retry

‎azure-functions-durable/CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ ADDED
1313

1414
FIXED
1515

16+
- With the corresponding core `durabletask` SDK fix, shared or duplicate pending
17+
tasks no longer strand native `when_all` or compatibility `context.task_all()`
18+
groups. This correction can change shared nested `when_any` / `context.task_any()`
19+
race winners during replay; see the [core changelog](../CHANGELOG.md) for upgrade
20+
guidance. Dependency minimums will be coordinated after the core fix is released.
1621
- With the corresponding core `durabletask` SDK fix, single-instance
1722
`purge_orchestration()` requests from `DurableFunctionsClient` and
1823
`SyncDurableFunctionsClient` now explicitly target orchestrations rather than

‎docs/supported-patterns.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,31 @@ def orchestrator(ctx: task.OrchestrationContext, _):
5151

5252
See the full [fan-out sample](../examples/fanout_fanin.py).
5353

54+
Tasks can be shared between `when_all` and `when_any` groups without scheduling
55+
the activity again. `when_all` preserves input order and every duplicate position:
56+
`when_all([work, work])` returns the result twice, whether `work` is pending or
57+
already complete. If a child fails, `when_all` waits for every child before raising
58+
the first observed failure.
59+
60+
`when_any` returns its winning direct child, including when that child is itself
61+
a composite. It stops observing losing children but does not cancel them or
62+
remove other groups' subscriptions. Completion notifications follow group
63+
subscription order, so shared nested races can select a composite rather than
64+
the leaf task that triggered it.
65+
66+
> [!WARNING]
67+
> Corrected shared-task notifications can change race winners during replay of
68+
> existing instances. Drain affected instances on the previous SDK or isolate
69+
> new instances in a separate task hub before upgrading. See the
70+
> [core changelog](../CHANGELOG.md) for migration guidance.
71+
72+
A canceled task still raises `TaskCancelledError` when its result is read.
73+
If collecting `when_all` results throws, that group does not notify its own
74+
parents; do not catch the error and reuse the group. Other groups observing
75+
the same child are still notified. A single completion callback error is
76+
re-raised unchanged; multiple callback errors are collected in one exception
77+
whose `errors` tuple retains each original exception.
78+
5479
## Human interaction and durable timers
5580

5681
An orchestration can wait for a user-defined event, such as a human approval event, before proceeding

‎durabletask-azuremanaged/CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ ADDED
1313

1414
FIXED
1515

16+
- With the corresponding core `durabletask` SDK fix, shared or duplicate pending
17+
tasks no longer leave `when_all` groups waiting after their children complete.
18+
This correction can change shared nested race winners during replay; see the
19+
[core changelog](../CHANGELOG.md) for upgrade guidance. Dependency minimums will
20+
be coordinated after the core fix is released.
1621
- With the corresponding core `durabletask` SDK fix, single-instance
1722
`purge_orchestration()` requests from synchronous and asynchronous Azure Managed
1823
clients now explicitly target orchestrations rather than entities.

‎durabletask/task.py‎

Lines changed: 58 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -608,17 +608,28 @@ class TaskCancelledError(Exception):
608608
"""Exception type for cancelled orchestration tasks."""
609609

610610

611+
class _TaskCompletionError(Exception):
612+
"""Collect callback errors without requiring Python 3.11 ExceptionGroup."""
613+
614+
def __init__(self, errors: list[Exception]) -> None:
615+
self.errors = tuple(errors)
616+
super().__init__(
617+
"Multiple task completion callbacks failed: "
618+
+ "; ".join(f"{type(error).__name__}: {error}" for error in errors))
619+
620+
611621
class Task(ABC, Generic[T]):
612622
"""Abstract base class for asynchronous tasks in a durable orchestration."""
613623
_result: T
614624
_exception: TaskFailedError | None
615-
_parent: CompositeTask[Any] | None
625+
_parents: dict[int, CompositeTask[Any]]
616626

617627
def __init__(self) -> None:
618628
super().__init__()
619629
self._is_complete = False
620630
self._exception = None
621-
self._parent = None
631+
# Dict insertion order preserves subscription order for shared races.
632+
self._parents = {}
622633

623634
@property
624635
def is_complete(self) -> bool:
@@ -650,8 +661,19 @@ def get_exception(self) -> TaskFailedError:
650661
return self._exception
651662

652663
def _notify_parent(self) -> None:
653-
if self._parent is not None:
654-
self._parent.on_child_completed(self)
664+
parents = tuple(self._parents.values())
665+
self._parents.clear()
666+
errors: list[Exception] = []
667+
for parent in parents:
668+
try:
669+
parent.on_child_completed(self)
670+
except Exception as error:
671+
# A canceled result can throw without blocking other observers.
672+
errors.append(error)
673+
if len(errors) == 1:
674+
raise errors[0]
675+
if errors:
676+
raise _TaskCompletionError(errors) from errors[0]
655677

656678

657679
class CompositeTask(Task[T]):
@@ -665,10 +687,24 @@ def __init__(self, tasks: list[Task[Any]]):
665687
self._tasks = tasks
666688
self._completed_tasks = 0
667689
self._failed_tasks = 0
690+
# Track references by identity, including unhashable Task subclasses.
691+
self._pending_children: dict[int, tuple[Task[Any], int]] = {}
668692
for task in tasks:
669-
task._parent = self
693+
_, count = self._pending_children.get(id(task), (task, 0))
694+
self._pending_children[id(task)] = (task, count + 1)
695+
for task, _ in tuple(self._pending_children.values()):
696+
if self.is_complete:
697+
break
670698
if task.is_complete:
671699
self.on_child_completed(task)
700+
else:
701+
task._parents[id(self)] = self
702+
703+
def _notify_parent(self) -> None:
704+
for task, _ in self._pending_children.values():
705+
task._parents.pop(id(self), None)
706+
self._pending_children.clear()
707+
super()._notify_parent()
672708

673709
def get_tasks(self) -> list[Task[Any]]:
674710
return self._tasks
@@ -698,10 +734,14 @@ def pending_tasks(self) -> int:
698734

699735
def on_child_completed(self, task: Task[Any]) -> None:
700736
if self.is_complete:
701-
raise ValueError('The task has already completed.')
702-
self._completed_tasks += 1
737+
return
738+
pending = self._pending_children.pop(id(task), None)
739+
if pending is None:
740+
return
741+
_, occurrences = pending
742+
self._completed_tasks += occurrences
703743
if task.is_failed:
704-
self._failed_tasks += 1
744+
self._failed_tasks += occurrences
705745
if self._pending_exception is None:
706746
# Stage the first failure but do NOT expose it via `_exception`
707747
# yet. Exposing it now would make `is_failed` return True while
@@ -871,12 +911,20 @@ def on_child_completed(self, task: Task[Any]) -> None:
871911

872912

873913
def when_all(tasks: list[Task[T]]) -> WhenAllTask[T]:
874-
"""Returns a task that completes when all of the provided tasks complete or when one of the tasks fail."""
914+
"""Wait for every provided task to complete or fail.
915+
916+
Results preserve input order and duplicate task references. Tasks may be
917+
shared between groups without scheduling additional work. If any task fails,
918+
the first observed failure is raised after every task has finished.
919+
"""
875920
return WhenAllTask(tasks)
876921

877922

878923
def when_any(tasks: Sequence[Task[T]]) -> WhenAnyTask[T]:
879-
"""Returns a task that completes when any of the provided tasks complete or fail."""
924+
"""Return the first direct child to complete or fail.
925+
926+
Tasks may be shared between groups. Losing tasks are not canceled.
927+
"""
880928
return WhenAnyTask(list(tasks))
881929

882930

‎tests/azure-functions-durable/test_orchestration_context_compat.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import pytest
88

9+
from durabletask import task
910
from azure.durable_functions.internal.compat.orchestration_context import (
1011
DurableOrchestrationContext,
1112
accepts_two_positional_args,
@@ -29,6 +30,25 @@ def test_get_input_returns_stored_input():
2930
assert adapter.get_input() == {"x": 1}
3031

3132

33+
def test_task_helpers_support_shared_and_duplicate_pending_tasks():
34+
adapter, _ = _adapter()
35+
a = task.CompletableTask[str]()
36+
b = task.CompletableTask[str]()
37+
c = task.CompletableTask[str]()
38+
left = adapter.task_all([a, b, a])
39+
right = adapter.task_all([a, c])
40+
race = adapter.task_any([a, c])
41+
root = adapter.task_all([left, right])
42+
43+
a.complete("A")
44+
b.complete("B")
45+
c.complete("C")
46+
47+
assert root.is_complete
48+
assert root.get_result() == [["A", "B", "A"], ["A", "C"]]
49+
assert race.get_result() is a
50+
51+
3252
def test_property_delegation():
3353
adapter, fake = _adapter()
3454
assert adapter.instance_id == "iid"

‎tests/durabletask/test_orchestration_e2e.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,59 @@ def sequence(ctx: task.OrchestrationContext, start_val: int):
8888
assert state.serialized_custom_status is None
8989

9090

91+
@pytest.mark.parametrize("pattern, expected, invocations", [
92+
("shared", [["A", "B"], ["A", "C"]], ["A", "B", "C"]),
93+
("duplicate", ["V", "V"], ["V"]),
94+
("precompleted_duplicate", ["V", "V"], ["V"]),
95+
("distinct", ["A", "B", "C"], ["A", "B", "C"]),
96+
("nested_disjoint", [["A", "B"], ["C"]], ["A", "B", "C"]),
97+
])
98+
def test_task_composition_executes_each_activity_once(pattern, expected, invocations):
99+
calls: list[str] = []
100+
lock = threading.Lock()
101+
102+
def echo(_: task.ActivityContext, value: str) -> str:
103+
with lock:
104+
calls.append(value)
105+
return value
106+
107+
def orchestrator(ctx: task.OrchestrationContext, _):
108+
if pattern in ("duplicate", "precompleted_duplicate"):
109+
work = ctx.call_activity(echo, input="V")
110+
if pattern == "precompleted_duplicate":
111+
yield work
112+
return (yield task.when_all([work, work]))
113+
114+
a = ctx.call_activity(echo, input="A")
115+
b = ctx.call_activity(echo, input="B")
116+
c = ctx.call_activity(echo, input="C")
117+
if pattern == "shared":
118+
left = task.when_all([a, b])
119+
right = task.when_all([a, c])
120+
return (yield task.when_all([left, right]))
121+
if pattern == "nested_disjoint":
122+
return (yield task.when_all([task.when_all([a, b]), task.when_all([c])]))
123+
return (yield task.when_all([a, b, c]))
124+
125+
with worker.TaskHubGrpcWorker(host_address=HOST) as w:
126+
w.add_orchestrator(orchestrator)
127+
w.add_activity(echo)
128+
w.start()
129+
with client.TaskHubGrpcClient(host_address=HOST) as c:
130+
instance_id = c.schedule_new_orchestration(orchestrator)
131+
state = c.wait_for_orchestration_completion(instance_id, timeout=10)
132+
committed_history = c.get_orchestration_history(instance_id)
133+
134+
assert state is not None
135+
assert state.runtime_status == client.OrchestrationStatus.COMPLETED
136+
assert state.serialized_output == json.dumps(expected)
137+
assert sorted(calls) == invocations
138+
assert sum(isinstance(event, history.TaskScheduledEvent)
139+
for event in committed_history) == len(invocations)
140+
assert sum(isinstance(event, history.TaskCompletedEvent)
141+
for event in committed_history) == len(invocations)
142+
143+
91144
def test_activity_error_handling():
92145

93146
def throw(_: task.ActivityContext, input: int) -> int:

‎tests/durabletask/test_orchestration_executor.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3117,7 +3117,8 @@ def orchestrator(ctx: task.OrchestrationContext, _):
31173117
assert complete_action.result.value == encoded_output
31183118

31193119

3120-
def test_when_all_with_retry():
3120+
@pytest.mark.parametrize("shared", [False, True])
3121+
def test_when_all_with_retry(shared):
31213122
"""Tests that a when_all pattern works correctly with retries"""
31223123
def dummy_activity(ctx, inp: str):
31233124
if inp == "Tokyo":
@@ -3134,7 +3135,12 @@ def orchestrator(ctx: task.OrchestrationContext, _):
31343135
retry_timeout=timedelta(seconds=50)),
31353136
input="Tokyo")
31363137
t2 = ctx.call_activity(dummy_activity, input="Seattle")
3137-
results = yield task.when_all([t1, t2])
3138+
if shared:
3139+
left = task.when_all([t1, t2, t1])
3140+
right = task.when_all([t1, t2])
3141+
results = yield task.when_all([left, right])
3142+
else:
3143+
results = yield task.when_all([t1, t2])
31383144
return results
31393145

31403146
registry = worker._Registry()

0 commit comments

Comments
 (0)