Skip to content

feat(dynaflow): add per-node retry policy, error fallback edges & state resumability - #338

Open
Kaap10 wants to merge 1 commit into
codeforstartups:developmentfrom
Kaap10:feat/graph-engine-resilience
Open

Kaap10 wants to merge 1 commit into
codeforstartups:developmentfrom
Kaap10:feat/graph-engine-resilience

Conversation

@Kaap10

@Kaap10 Kaap10 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR implements fault tolerance, error recovery routing, and checkpoint resumability for the dynaflow graph execution engine.

Production agent workflows must survive transient API outages and support recovery branches without crashing or losing progress. This PR introduces per-node retry policies, fallback error edges, node lifecycle tracking, and zero-loss state resumption.

Related issue

Fixes #279

Changes

  • RetryPolicy (dynavec.nodes.base):
    • Configurable exponential backoff with full jitter (max_attempts, base_delay, max_delay, jitter).
    • Customizable exception filters (retry_on).
    • Serialization / deserialization (to_dict / from_dict).
  • NodeStatus & Lifecycle:
    • Real-time node state tracking: "pending", "running", "retrying", "succeeded", "failed", "skipped".
    • Node instance attributes: attempts, last_error, status.
  • Built-in Nodes (dynavec.nodes.builtin):
    • Updated all 9 built-in nodes (RetrieverNode, LLMNode, ToolNode, RouterNode, CodeNode, ApprovalNode, LoopNode, WebhookNode, SubGraphNode) to support retry_policy.
  • Error Fallback Edges (dynavec.nodes.graph):
    • Added Edge.is_error_edge and SimpleGraph.add_error_edge() to route from failed nodes to recovery handlers.
  • GraphRunState & Resumability (SimpleGraph.resume):
    • Serializable state checkpoints (run_id, node_outputs, node_statuses, node_errors, completed, failed_node).
    • Resumes execution of partially-completed or failed runs without re-executing previously succeeded nodes.
  • Type-safe @overload:
    • SimpleGraph.run() returns dict[str, Any] by default (100% backwards compatible) or GraphRunState when return_state=True.

Testing

  • 21/21 node tests pass locally (pytest tests/test_nodes.py)
  • Full regression suite passes (pytest tests/)
  • Ruff checks pass
  • Strict MyPy type checks pass (0 errors)

Checklist

  • My changes are focused and relevant to this pull request.
  • I have added or updated tests where appropriate.
  • I have reviewed my changes for unrelated modifications.
  • I have updated documentation where necessary.

…te resumability (closes codeforstartups#279)

Adds configurable RetryPolicy with exponential backoff and full jitter.

Adds Edge.is_error_edge and SimpleGraph.add_error_edge() for graceful error recovery routing.

Adds GraphRunState checkpointing and SimpleGraph.resume() to continue partially-failed runs without duplicate execution.

Adds real-time NodeStatus lifecycle tracking across all built-in node types.
@Kaap10

Kaap10 commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

up for the review!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Engine: retries, error handling & resumability

1 participant