From 7aa424dda134ca40e8953597e88091bc1ff69c36 Mon Sep 17 00:00:00 2001 From: Markus Date: Sat, 26 Sep 2026 16:24:11 +0200 Subject: [PATCH 01/28] feat(workflows): compose workflows with a unified execution tree Keep invocation results and workflow bindings on the same execution occurrence. Isolate fan-out contexts and resume persisted expansions through one executor. Assisted-by: OpenCode (model: gpt-6-astra, autonomous) --- design/workflow-step.md | 7 +- docs/reference/workflows.md | 67 ++ src/specify_cli/workflows/__init__.py | 2 + src/specify_cli/workflows/_commands.py | 27 +- src/specify_cli/workflows/_execution.py | 576 +++++++++++++++ src/specify_cli/workflows/command_resume.py | 2 +- src/specify_cli/workflows/command_status.py | 9 + src/specify_cli/workflows/composition.py | 147 ++++ src/specify_cli/workflows/engine.py | 511 +++---------- .../workflows/step/gate/__init__.py | 2 + .../workflows/step/workflow/__init__.py | 17 + .../workflows/test_command_status.py | 32 + tests/workflows/test_composition_execution.py | 670 ++++++++++++++++++ workflows/ARCHITECTURE.md | 13 +- workflows/PUBLISHING.md | 2 +- workflows/README.md | 5 +- 16 files changed, 1655 insertions(+), 434 deletions(-) create mode 100644 src/specify_cli/workflows/_execution.py create mode 100644 src/specify_cli/workflows/composition.py create mode 100644 src/specify_cli/workflows/step/workflow/__init__.py create mode 100644 tests/workflows/test_composition_execution.py diff --git a/design/workflow-step.md b/design/workflow-step.md index e9ff094298..ddb902d840 100644 --- a/design/workflow-step.md +++ b/design/workflow-step.md @@ -41,9 +41,10 @@ unless `continue_on_error: true` is set (an explicit abort always stops). The engine calls `validate()` during workflow validation but does not automatically validate a definition passed to `execute()`. Guard invalid configurations in `execute()` too, returning a failed result rather than a -successful default or an unhandled exception. Resume restarts the current -top-level step; a pause inside nested steps re-runs their parent and nested -body. Design side effects accordingly. +successful default or an unhandled exception. New runs persist an execution +tree: resume retries unfinished occurrences and restores completed results and +selected expansions. Legacy states enter through their old top-level index. +Side effects performed before their completion checkpoint can still repeat. The registry holds one shared instance per type. Concurrent `fan-out` can invoke that instance from multiple threads: keep execution stateless and diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 811ab4ebf4..c27ece7342 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -540,6 +540,73 @@ specify workflow run speckit -i spec="Build a kanban board with drag-and-drop ta | `do-while` | Execute at least once, then loop on condition | | `fan-out` | Dispatch a step for each item in a list | | `fan-in` | Aggregate results from a fan-out step | +| `workflow` | Call an installed workflow with private inputs and declared outputs | + +### Workflow composition + +A `workflow` step executes an installed, enabled workflow in the current project +as a private scope of the same run. Targets can be literal IDs or expressions; +the resolved string must match the ID exactly, including case and whitespace. + +```yaml +inputs: + target: {type: string, required: true} + report: {type: string, required: true} +steps: + - id: investigate + type: workflow + workflow: "{{ inputs.target }}" + input: + report: "{{ inputs.report }}" +``` + +The included workflow sees only declared inputs passed through `input` and its +own step results. Unknown input names, missing required values, and invalid +types/enums fail the call. Existing defaults and `integration: auto` resolution +apply. Values cross back only through explicit declarations: + +```yaml +outputs: + report: + value: "{{ steps.analyze.output.stdout }}" +``` + +The caller reads `{{ steps.investigate.output.report }}`. Output includes +`workflow` and `status`; failures also include `error`, and an abort includes +`aborted: true`. These names and `integration`, `model`, `options`, and `input` +are reserved. Returned values must be JSON-safe. Private inputs, step records, +and logs are not part of the return value. No separate child run is created. + +Failures may be handled with `continue_on_error: true` at either the included +step or workflow-call boundary. Pauses and explicit aborts always stop execution. +Nested composition is allowed, but repeated workflow IDs on the active call +path are cycles. Diamonds are allowed. The maximum included depth is 16, with +the root at depth zero. + +### Execution identity and resume + +Each step occurrence owns a record in a persisted execution tree. Authored step +names are local expression aliases, not global execution IDs. Fan-out items +have independent alias contexts; results remain ordered by item index. + +New runs persist selected branches, dynamic custom-step expansions, loop +iterations, and fan-out items. Resume retries unfinished operations and retains +completed work without reevaluating already selected branches. Workflow targets +and overlay-resolved definitions remain bound even if installations change. +Ordinary resume retains bound inputs; explicit `--input` updates rebind reached, +incomplete calls through their original mappings. Completed calls retain their +results. Failed output evaluation retries finalization without repeating child +commands. + +Snapshots are stored as YAML strings inside the private JSON execution tree, +preserving YAML scalar types. Inputs and results remain JSON values. Legacy runs +without a tree enter through their saved top-level index, then use tree-backed +resume. A tree-backed run left `running` by a crashed process can also be resumed; +only one process may execute or resume a run at a time. A side effect completed +before its checkpoint may execute again after a crash. + +Run/resume/status JSON includes `workflow_scopes` summaries when calls exist and +reports the active nested gate with its `scope_path`. > **Security note:** a `shell` step runs a local command with **your** privileges. There is no capability sandbox — `requires` is an advisory pre-condition block (spec-kit version, integrations), not a runtime gate, so it does **not** restrict what a step can do. In particular there is no `requires.permissions` capability gate: it is rejected by validation precisely because it would imply a sandbox that does not exist. Review any catalog or downloaded workflow before running it, and use a `gate` step to require explicit approval before sensitive or destructive shell commands. diff --git a/src/specify_cli/workflows/__init__.py b/src/specify_cli/workflows/__init__.py index 2bb3de56a5..dca51280e5 100644 --- a/src/specify_cli/workflows/__init__.py +++ b/src/specify_cli/workflows/__init__.py @@ -49,6 +49,7 @@ def _register_builtin_steps() -> None: from .step.fan_in import FanInStep from .step.fan_out import FanOutStep from .step.gate import GateStep + from .step.workflow import WorkflowStep from .step.if_then import IfThenStep from .step.init import InitStep from .step.prompt import PromptStep @@ -62,6 +63,7 @@ def _register_builtin_steps() -> None: _register_step(FanInStep()) _register_step(FanOutStep()) _register_step(GateStep()) + _register_step(WorkflowStep()) _register_step(IfThenStep()) _register_step(InitStep()) _register_step(PromptStep()) diff --git a/src/specify_cli/workflows/_commands.py b/src/specify_cli/workflows/_commands.py index de01789d0c..7d5757504c 100644 --- a/src/specify_cli/workflows/_commands.py +++ b/src/specify_cli/workflows/_commands.py @@ -940,6 +940,12 @@ def _workflow_run_payload(state: Any) -> dict[str, Any]: error = _failed_step_error(state) if error is not None: payload["error"] = error + if getattr(state, "execution", None): + from ._execution import scope_summaries + + scopes = scope_summaries(state.execution) + if scopes: + payload["workflow_scopes"] = scopes return payload @@ -978,21 +984,36 @@ def _gate_outcome(state: Any) -> dict[str, Any] | None: if getattr(state.status, "value", state.status) not in ("paused", "aborted"): return None step = (getattr(state, "step_results", None) or {}).get(state.current_step_id) + step_id = state.current_step_id + scope_path = None + if getattr(state, "execution", None): + from ._execution import active_step + + active = active_step(state.execution) + if active is None: + return None + path, node = active + step_id = path[-1] + scope_path = path[:-1] + step = node.get("result") if not isinstance(step, dict) or not _is_gate_step(step): return None output = step.get("output") or {} # `message`, `options`, and `choice` may be non-string YAML literals in an - # unvalidated workflow (GateStep coerces none of them for the payload), so + # legacy or synthetic records, so # normalise all three for a stable JSON schema: message → str, options → # list[str] | None, choice → str | None (None means no decision yet). message = output.get("message") choice = output.get("choice") - return { - "step_id": state.current_step_id, + detail = { + "step_id": step_id, "message": None if message is None else str(message), "options": _normalize_gate_options(output.get("options")), "choice": None if choice is None else str(choice), } + if scope_path: + detail["scope_path"] = scope_path + return detail def _normalize_gate_options(options: Any) -> list[str] | None: diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py new file mode 100644 index 0000000000..7f540af0b9 --- /dev/null +++ b/src/specify_cli/workflows/_execution.py @@ -0,0 +1,576 @@ +"""A persisted execution tree, traversed identically on first execution and resume. + +An occurrence owns its result and descendants. Authored names are only aliases +in an expression context; tree positions distinguish repeated executions. +""" + +from __future__ import annotations + +from concurrent.futures import ThreadPoolExecutor +from copy import deepcopy +from dataclasses import replace +import threading +from typing import Any + +import yaml + +from .base import StepContext, StepResult, StepStatus +from .composition import bind_inputs, evaluate_outputs, resolve_target, validate_call +from .expressions import evaluate_condition, evaluate_expression + +HALTING = {"paused", "failed", "aborted"} + + +class CheckpointError(RuntimeError): + """Persistence failed; reload the authoritative disk checkpoint before retrying.""" + + +def sequence(steps: list[dict[str, Any]]) -> dict[str, Any]: + return { + "source": yaml.safe_dump(steps, sort_keys=False), + "nodes": [{"phase": "ready"} for _ in steps], + } + + +def validate_execution(tree: Any) -> None: + """Validate stored structure without importing a project's custom steps.""" + + def check_sequence(seq): + if not isinstance(seq, dict) or not isinstance(seq.get("source"), str): + raise ValueError("Invalid execution sequence") + steps = yaml.safe_load(seq["source"]) + nodes = seq.get("nodes") + if ( + not isinstance(steps, list) + or not all(isinstance(s, dict) for s in steps) + or not isinstance(nodes, list) + or len(nodes) != len(steps) + ): + raise ValueError("Invalid execution sequence length or steps") + for step, node in zip(steps, nodes): + if not isinstance(node, dict) or node.get("phase") not in { + "ready", + "children", + "outputs", + "blocked", + "done", + }: + raise ValueError("Invalid execution phase") + result = node.get("result") + if result is not None and ( + not isinstance(result, dict) + or result.get("status") not in {s.value for s in StepStatus} + or not isinstance(result.get("output"), dict) + ): + raise ValueError("Invalid execution result") + if node["phase"] == "done" and result is None: + raise ValueError("Completed execution lacks a result") + if node.get("outcome", "completed") not in {"completed", *HALTING}: + raise ValueError("Invalid execution outcome") + if ( + node["phase"] == "done" + and node.get("outcome", "completed") != "completed" + ): + raise ValueError("Completed execution has a blocking outcome") + if node["phase"] == "blocked" and ( + result is None + or result["status"] not in {"failed", "paused"} + or node.get("outcome") not in HALTING + ): + raise ValueError("Blocked execution lacks a blocking result") + children = node.get("children", []) + if not isinstance(children, list): + raise ValueError("Invalid execution children") + for child in children: + check_sequence(child) + binding = node.get("binding") + if binding is not None: + if step.get("type") != "workflow" or not isinstance(binding, dict): + raise ValueError("Invalid workflow binding") + definition = yaml.safe_load(binding.get("definition", "")) + if ( + not isinstance(definition, dict) + or not isinstance(definition.get("workflow"), dict) + or definition["workflow"].get("id") != binding.get("workflow") + or not isinstance(binding.get("inputs"), dict) + or len(children) != 1 + or not isinstance(definition.get("steps"), list) + or definition["steps"] != yaml.safe_load(children[0]["source"]) + ): + raise ValueError("Invalid bound workflow definition or inputs") + if node["phase"] == "outputs" and binding is None: + raise ValueError("Output finalization requires a workflow binding") + if node["phase"] == "children" and ( + not children or (binding is None and result is None) + ): + raise ValueError("Expanded execution lacks its children or result") + if node["phase"] == "ready" and (result is not None or children or binding): + raise ValueError("Unstarted execution already has progress") + if step.get("type") == "fan-out" and children: + items = (result or {}).get("output", {}).get("items") + if not isinstance(items, list) or len(items) != len(children): + raise ValueError("Fan-out items do not match execution children") + + try: + if not isinstance(tree, dict) or tree.get("version") != 1: + raise ValueError("Unsupported execution version") + offset = tree.get("offset", 0) + if ( + type(offset) is not int + or offset < 0 + or not isinstance(tree.get("initial", {}), dict) + ): + raise ValueError("Invalid execution offset or initial aliases") + check_sequence(tree["sequence"]) + except (KeyError, TypeError, yaml.YAMLError, RecursionError) as exc: + raise ValueError(f"Invalid execution state: {exc}") from exc + + +def active_step(tree): + """First unfinished leaf in execution order, including nested workflow scopes.""" + + def walk(seq, path): + for index, (config, node) in enumerate( + zip(yaml.safe_load(seq["source"]), seq["nodes"]) + ): + if node["phase"] == "done": + continue + here = [*path, config.get("id", f"step-{index}")] + for item, child in enumerate(node.get("children", [])): + child_path = ( + [*here, str(item)] + if config.get("type") in {"fan-out", "while", "do-while"} + else here + ) + found = walk(child, child_path) + if found: + return found + return here, node + return None + + return walk(tree["sequence"], []) + + +def scope_summaries(tree): + """Report workflow boundaries without exposing private inputs or results.""" + summaries = [] + + def walk(seq, path): + for index, (config, node) in enumerate( + zip(yaml.safe_load(seq["source"]), seq["nodes"]) + ): + here = [*path, config.get("id", f"step-{index}")] + binding = node.get("binding") + if binding: + output = node.get("result", {}).get("output", {}) + summaries.append( + { + "scope_path": here, + "workflow_id": binding["workflow"], + "status": output.get("status", "running"), + } + ) + for item, child in enumerate(node.get("children", [])): + child_path = ( + [*here, str(item)] + if config.get("type") in {"fan-out", "while", "do-while"} + else here + ) + walk(child, child_path) + + walk(tree["sequence"], []) + return summaries + + +class Execution: + def __init__(self, engine, state, registry, *, rebind=False): + self.engine, self.state, self.registry = engine, state, registry + self.rebind = rebind + + def commit(self, node=None, changes=None, *, context=None, name=None, public=None): + """Mutate an occurrence and its compatibility views in one checkpoint.""" + with self.state._lock: + if self.state._checkpoint_failed: + raise CheckpointError("A previous checkpoint failed") + if node is not None: + node.update(changes or {}) + if context is not None and "result" in node: + context.steps[name] = node["result"] + if public is not None: + self.state.step_results[public] = node["result"] + self.state.save() + + def log(self, event, name, path, workflow, **fields): + entry = {"event": event, "step_id": name, **fields} + if path: + entry.update(execution_path=list(path), workflow_id=workflow) + self.state.append_log(entry) + + def run( + self, + seq, + context, + ancestry, + *, + path=(), + public=True, + root=False, + loop_alias=None, + ): + steps = yaml.safe_load(seq["source"]) + for index, (config, node) in enumerate(zip(steps, seq["nodes"])): + config = {"id": f"step-{index}", **config} + name = config["id"] + occurrence = (*path, index) + public_name = name if public else None + if root and node["phase"] != "done": + with self.state._lock: + self.state.current_step_index = index + self.state.execution.get( + "offset", 0 + ) + self.state.current_step_id = name + outcome = self.step( + config, node, context, ancestry, occurrence, public_name + ) + if loop_alias is not None and "result" in node: + with self.state._lock: + key = f"{loop_alias[0]}:{name}:{loop_alias[1]}" + self.state.step_results[key] = node["result"] + context.steps[key] = node["result"] + self.state.save() + if outcome in HALTING: + return outcome, node.get("error") + return "completed", None + + def restore(self, node, context): + """Rebuild aliases for completed expansions without re-executing them.""" + for child in node.get("children", []) if "binding" not in node else []: + for index, (config, nested) in enumerate( + zip(yaml.safe_load(child["source"]), child["nodes"]) + ): + if "result" in nested: + context.steps[config.get("id", f"step-{index}")] = nested["result"] + if config.get("type") != "fan-out": + self.restore(nested, context) + + def step(self, config, node, context, ancestry, path, public_name): + name = config.get("id", "step-0") + kind = config.get("type", "command") + if node["phase"] == "done": + context.steps[name] = node["result"] + if kind == "fan-out": + template = node["result"]["output"].get("step_template", {}) + for index, child in enumerate(node.get("children", [])): + result = child["nodes"][0].get("result") + if result is not None: + context.steps[ + f"{name}:{template.get('id', 'item')}:{index}" + ] = result + else: + self.restore(node, context) + return node.get("outcome", "completed") + if node.get("outcome") == "aborted": + return "aborted" + + if kind == "workflow": + return self.workflow(config, node, context, ancestry, path, public_name) + + if node["phase"] in {"ready", "blocked"}: + with self.state._lock: + self.state.current_step_id = name + self.log("step_started", name, path, ancestry[-1], type=kind) + if self.engine.on_step_start is not None: + with self.engine._callback_lock: + self.engine.on_step_start(name, config.get("command", "") or kind) + impl = self.registry.get(kind) + result = ( + impl.execute(config, context) + if impl + else StepResult(StepStatus.FAILED, error=f"Unknown step type: {kind!r}") + ) + if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: + return self.finish( + config, node, result, context, ancestry, path, public_name + ) + children = [sequence(result.next_steps)] if result.next_steps else [] + if kind == "fan-out": + template = result.output.get("step_template", {}) + children = ( + [sequence([template]) for _ in result.output.get("items", [])] + if template + else [] + ) + data = self.record(config, result, context) + if not children: + if kind == "fan-out": + result.output = {**result.output, "results": []} + return self.finish( + config, node, result, context, ancestry, path, public_name + ) + self.commit( + node, + {"phase": "children", "result": data, "children": children}, + context=context, + name=name, + public=public_name, + ) + else: + context.steps[name] = node["result"] + + if kind == "fan-out": + outcome, error, outputs = self.fan_out( + config, node, context, ancestry, path, public_name + ) + data = { + **node["result"], + "output": {**node["result"]["output"], "results": outputs}, + } + self.commit( + node, {"result": data}, context=context, name=name, public=public_name + ) + else: + outcome, error = "completed", None + for iteration, child in enumerate(node["children"]): + outcome, error = self.run( + child, + context, + ancestry, + path=(*path, iteration), + public=public_name is not None, + loop_alias=(name, iteration) + if kind in {"while", "do-while"} + and iteration + and public_name is not None + else None, + ) + if outcome in HALTING: + break + if outcome == "completed" and kind in {"while", "do-while"}: + limit = config.get("max_iterations", 10) + if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1: + limit = 10 + while len(node["children"]) < limit and evaluate_condition( + config.get("condition", False), context + ): + child = sequence(yaml.safe_load(node["children"][0]["source"])) + self.commit(node, {"children": [*node["children"], child]}) + outcome, error = self.run( + child, + context, + ancestry, + path=(*path, len(node["children"]) - 1), + public=public_name is not None, + loop_alias=(name, len(node["children"]) - 1) + if public_name is not None + else None, + ) + if outcome in HALTING: + break + self.commit( + node, + { + "phase": "done" if outcome == "completed" else "children", + "outcome": outcome, + "error": error, + }, + ) + return outcome + + @staticmethod + def record(config, result, context): + output = result.output + data = { + "type": config.get("type", "command"), + "integration": output.get("integration") + or config.get("integration") + or context.default_integration, + "model": output.get("model") + or config.get("model") + or context.default_model, + "options": output.get("options") or config.get("options", {}), + "input": {} + if config.get("type") == "workflow" + else output.get("input") or config.get("input", {}), + "output": output, + "status": result.status.value, + "error": result.error, + } + if data["type"] == "command" and "integration_args" in output: + data.update( + integration_args=output["integration_args"], + integration_options=output["integration_options"], + ) + return data + + def finish(self, config, node, result, context, ancestry, path, public_name): + name = config.get("id", "step-0") + outcome = "completed" + if result.status == StepStatus.PAUSED: + outcome = "paused" + elif result.status == StepStatus.FAILED: + outcome = "aborted" if result.output.get("aborted") else "failed" + if outcome == "failed" and config.get("continue_on_error") is True: + outcome = "completed" + self.commit( + node, + { + "phase": "done" if outcome == "completed" else "blocked", + "result": self.record(config, result, context), + "outcome": outcome, + "error": result.error, + }, + context=context, + name=name, + public=public_name, + ) + self.log("step_completed", name, path, ancestry[-1], status=result.status.value) + if result.status == StepStatus.FAILED: + event = { + "aborted": "workflow_aborted", + "completed": "step_continue_on_error", + }.get(outcome, "step_failed") + self.log(event, name, path, ancestry[-1], error=result.error) + return outcome + + def workflow(self, config, node, context, ancestry, path, public_name): + from .engine import WorkflowDefinition + + name = config.get("id", "step-0") + self.log("step_started", name, path, ancestry[-1], type="workflow") + if self.engine.on_step_start is not None: + with self.engine._callback_lock: + self.engine.on_step_start(name, "workflow") + binding = node.get("binding") + target = binding["workflow"] if binding else config.get("workflow") + try: + if binding is None: + errors = validate_call(config) + if errors: + raise ValueError("; ".join(errors)) + target = evaluate_expression(target, context) + definition = resolve_target(self.state.project_root, target, ancestry) + binding = { + "workflow": target, + "definition": yaml.safe_dump(definition.data, sort_keys=False), + "inputs": bind_inputs(self.engine, definition, config, context), + "workflow_dir": str(definition.source_path.parent) + if definition.source_path + else None, + } + self.commit( + node, + { + "binding": binding, + "children": [sequence(definition.steps)], + "phase": "children", + }, + ) + else: + definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) + if self.rebind: + binding = { + **binding, + "inputs": bind_inputs(self.engine, definition, config, context), + } + self.commit(node, {"binding": binding}) + child_context = StepContext( + inputs=binding["inputs"], + project_root=context.project_root, + run_id=context.run_id, + is_resume=context.is_resume, + workflow_dir=binding["workflow_dir"], + default_integration=definition.default_integration, + default_model=definition.default_model, + default_options=definition.default_options, + ) + outcome, error = self.run( + node["children"][0], + child_context, + (*ancestry, target), + path=(*path, "workflow"), + public=False, + ) + output = {"workflow": target, "status": outcome} + if outcome == "completed": + self.commit(node, {"phase": "outputs"}) + output.update(evaluate_outputs(definition, child_context)) + elif outcome == "aborted": + output["aborted"] = True + if error is not None: + output["error"] = error + status = ( + StepStatus.COMPLETED + if outcome == "completed" + else StepStatus.PAUSED + if outcome == "paused" + else StepStatus.FAILED + ) + result = StepResult(status, output=output, error=error) + except CheckpointError: + raise + except Exception as exc: + target = target if isinstance(target, str) else repr(target) + result = StepResult( + StepStatus.FAILED, + output={"workflow": target, "status": "failed", "error": str(exc)}, + error=str(exc), + ) + return self.finish(config, node, result, context, ancestry, path, public_name) + + def fan_out(self, config, node, context, ancestry, path, public_name): + output = node["result"]["output"] + items = output.get("items", []) + try: + workers = max(1, int(output.get("max_concurrency", 1))) + except (TypeError, ValueError, OverflowError): + workers = 1 + workers = min(workers, len(items)) + initial = deepcopy(context.steps) + halted = threading.Event() + + def run_item(index): + local = replace( + context, steps=deepcopy(initial), item=items[index], inside_fan_out=True + ) + child = node["children"][index] + outcome, error = self.run( + child, local, ancestry, path=(*path, "item", index), public=False + ) + if outcome in HALTING: + halted.set() + record = child["nodes"][0].get("result", {}) + template_name = output.get("step_template", {}).get("id", "item") + with self.state._lock: + key = f"{config['id']}:{template_name}:{index}" + context.steps[key] = record + if public_name is not None: + self.state.step_results[key] = record + self.state.save() + return outcome, error, record.get("output", {}) + + results = [] + if workers <= 1: + for index in range(len(items)): + outcome, error, value = run_item(index) + results.append(value) + if outcome in HALTING: + return outcome, error, results + return "completed", None, results + with ThreadPoolExecutor(max_workers=workers) as pool: + futures = {i: pool.submit(run_item, i) for i in range(workers)} + for index in range(len(items)): + try: + outcome, error, value = futures.pop(index).result() + except BaseException: + for future in futures.values(): + future.cancel() + raise + results.append(value) + if outcome in HALTING: + for future in futures.values(): + future.cancel() + return outcome, error, results + following = index + workers + if following < len(items) and not halted.is_set(): + futures[following] = pool.submit(run_item, following) + return "completed", None, results diff --git a/src/specify_cli/workflows/command_resume.py b/src/specify_cli/workflows/command_resume.py index 3be24f9582..9a50a6b50c 100644 --- a/src/specify_cli/workflows/command_resume.py +++ b/src/specify_cli/workflows/command_resume.py @@ -18,7 +18,7 @@ def workflow_resume( help="Emit the resume outcome as a single JSON object instead of formatted text.", ), ): - """Resume a paused or failed workflow run.""" + """Resume a paused, failed, or crash-interrupted workflow run.""" from . import load_custom_steps from .engine import RunState, WorkflowEngine diff --git a/src/specify_cli/workflows/command_status.py b/src/specify_cli/workflows/command_status.py index d9af4b48d6..f58c990698 100644 --- a/src/specify_cli/workflows/command_status.py +++ b/src/specify_cli/workflows/command_status.py @@ -90,6 +90,15 @@ def workflow_status( s, "white" ) cli.console.print(f" [{sc}]●[/{sc}] {step_id}: {s}") + if state.execution: + from ._execution import scope_summaries + + for scope in scope_summaries(state.execution): + path = " / ".join(scope["scope_path"]) + cli.console.print( + f" {cli._escape_markup(path)} → " + f"{cli._escape_markup(scope['workflow_id'])}: {scope['status']}" + ) else: runs = engine.list_runs() diff --git a/src/specify_cli/workflows/composition.py b/src/specify_cli/workflows/composition.py new file mode 100644 index 0000000000..02c1156177 --- /dev/null +++ b/src/specify_cli/workflows/composition.py @@ -0,0 +1,147 @@ +"""Typed workflow-call boundary; execution and persistence belong to the engine.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from .base import StepContext +from .expressions import evaluate_expression + +RESERVED_OUTPUT_NAMES = frozenset( + { + "workflow", + "status", + "error", + "aborted", + "integration", + "model", + "options", + "input", + } +) +MAX_COMPOSITION_DEPTH = 16 + + +def require_json(value: Any) -> None: + """Reject lossy/non-JSON values, including cycles, before checkpointing.""" + try: + encoded = json.dumps(value, allow_nan=False) + if json.loads(encoded) != value: + raise ValueError("JSON encoding changes the value") + except (TypeError, ValueError, RecursionError) as exc: + raise ValueError(f"Value is not JSON-safe: {exc}") from exc + + +def validate_call(config: dict[str, Any]) -> list[str]: + from .engine import _ID_PATTERN + from .overlay.schema import _RESERVED_WORKFLOW_IDS + + errors = [] + target = config.get("workflow") + if not isinstance(target, str): + errors.append("'workflow' must be a string") + elif "{{" not in target and ( + not _ID_PATTERN.fullmatch(target) or target in _RESERVED_WORKFLOW_IDS + ): + errors.append("'workflow' must be an exact, safe, non-reserved workflow ID") + mapping = config.get("input", {}) + if mapping is not None: + if not isinstance(mapping, dict) or any( + not isinstance(k, str) for k in mapping + ): + errors.append("'input' must be a mapping with string keys") + else: + try: + require_json(mapping) + except ValueError as exc: + errors.append(str(exc)) + return errors + + +def validate_outputs(outputs: Any) -> list[str]: + from .engine import _ID_PATTERN + + if not isinstance(outputs, dict): + return ["'outputs' must be a mapping"] + errors = [] + for name, entry in outputs.items(): + if not isinstance(name, str) or not _ID_PATTERN.fullmatch(name): + errors.append(f"Output {name!r} must be a safe identifier") + elif name in RESERVED_OUTPUT_NAMES: + errors.append(f"Output {name!r} is reserved") + if not isinstance(entry, dict) or set(entry) != {"value"}: + errors.append(f"Output {name!r} must contain exactly 'value'") + else: + try: + require_json(entry["value"]) + except ValueError as exc: + errors.append(f"Output {name!r}: {exc}") + return errors + + +def resolve_target(project_root: Path, target: Any, ancestry: tuple[str, ...]): + from .catalog import WorkflowRegistry + from .engine import _ID_PATTERN, validate_workflow + from .overlay import WorkflowResolver + from .overlay.schema import _RESERVED_WORKFLOW_IDS + + if not isinstance(target, str) or not _ID_PATTERN.fullmatch(target): + raise ValueError("Workflow target must be an exact safe workflow ID") + if target in _RESERVED_WORKFLOW_IDS: + raise ValueError(f"Workflow {target!r} is reserved") + if target in ancestry: + raise ValueError( + f"Workflow composition cycle: {' -> '.join((*ancestry, target))}" + ) + if len(ancestry) > MAX_COMPOSITION_DEPTH: + raise ValueError( + f"Workflow composition exceeds maximum depth {MAX_COMPOSITION_DEPTH}" + ) + metadata = WorkflowRegistry(project_root).get(target) + if not isinstance(metadata, dict): + raise ValueError(f"Workflow {target!r} is not installed") + if not metadata.get("enabled", True): + raise ValueError(f"Workflow {target!r} is disabled") + definition = WorkflowResolver(project_root).resolve(target) + if definition.id != target: + raise ValueError( + f"Workflow {target!r} resolves to mismatched ID {definition.id!r}" + ) + errors = validate_workflow(definition) + if errors: + raise ValueError(f"Invalid workflow {target!r}: {'; '.join(errors)}") + return definition + + +def bind_inputs(engine, definition, config: dict[str, Any], context: StepContext): + mapping = config.get("input") + if mapping is None: + mapping = {} + if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): + raise ValueError("'input' must be a mapping with string keys") + provided = { + key: evaluate_expression(value, context) for key, value in mapping.items() + } + require_json(provided) + unknown = provided.keys() - definition.inputs.keys() + if unknown: + raise ValueError( + f"Undeclared inputs for workflow {definition.id!r}: {sorted(unknown)}" + ) + resolved = engine._resolve_inputs(definition, provided) + require_json(resolved) + return resolved + + +def evaluate_outputs(definition, context: StepContext) -> dict[str, Any]: + errors = validate_outputs(definition.outputs) + if errors: + raise ValueError("; ".join(errors)) + output = { + name: evaluate_expression(entry["value"], context) + for name, entry in definition.outputs.items() + } + require_json(output) + return output diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index d81aae3212..d947f355c5 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -10,14 +10,12 @@ from __future__ import annotations -import dataclasses import json import os import re import tempfile import threading import uuid -from concurrent.futures import Future, ThreadPoolExecutor from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -28,7 +26,7 @@ default_integration_key, try_read_integration_json, ) -from .base import RunStatus, StepContext, StepResult, StepStatus +from .base import RunStatus, StepContext # -- Workflow Definition -------------------------------------------------- @@ -86,6 +84,7 @@ def __init__(self, data: dict[str, Any], source_path: Path | None = None) -> Non # Inputs self.inputs: dict[str, Any] = data.get("inputs", {}) + self.outputs: Any = data.get("outputs", {}) # Steps self.steps: list[dict[str, Any]] = data.get("steps", []) @@ -140,7 +139,7 @@ def _get_valid_step_types() -> set[str]: return set(STEP_REGISTRY.keys()) return { "command", "shell", "prompt", "gate", "if", "init", "slot", - "switch", "while", "do-while", "fan-out", "fan-in", + "switch", "while", "do-while", "fan-out", "fan-in", "workflow", } @@ -183,7 +182,9 @@ def validate_workflow(definition: WorkflowDefinition) -> list[str]: An empty list means the workflow is valid. """ - errors: list[str] = [] + from .composition import validate_outputs + + errors: list[str] = validate_outputs(definition.outputs) # -- Schema version --------------------------------------------------- # str() so an unquoted ``schema_version: 1.0`` (YAML float) is accepted — @@ -695,10 +696,14 @@ def __init__( self.current_step_index = 0 self.current_step_id: str | None = None self.step_results: dict[str, dict[str, Any]] = {} + self.execution: dict[str, Any] | None = None + self._checkpoint_failed = False # Guards step_results mutation and save() so a concurrent fan-out cannot # mutate the dict while save() is serializing it (which would raise # "dictionary changed size during iteration"). - self._lock = threading.Lock() + self._lock = threading.RLock() + # Reentrant so an execution transition can update views and call save() + # under the same lock. Step implementations run outside that lock. # Serializes append_log's list append + log.jsonl write so concurrent # fan-out workers cannot interleave or corrupt log lines. Kept separate # from _lock so frequent logging never contends with state saves; since @@ -723,6 +728,10 @@ def record_step_result(self, step_id: str, data: dict[str, Any]) -> None: fan-out). For a sequential run this is an uncontended lock. """ with self._lock: + from ._execution import CheckpointError + + if self._checkpoint_failed: + raise CheckpointError("A previous checkpoint failed; reload the run") self.step_results[step_id] = data def set_step_output(self, step_id: str, output: Any) -> None: @@ -749,6 +758,10 @@ def save(self) -> None: runs_dir.mkdir(parents=True, exist_ok=True) with self._lock: + if self._checkpoint_failed: + from ._execution import CheckpointError + + raise CheckpointError("A previous checkpoint failed; reload the run") # Stamp updated_at inside the lock so the timestamp matches the # snapshot this thread serializes (concurrent savers don't race it). self.updated_at = datetime.now(timezone.utc).isoformat() @@ -765,9 +778,18 @@ def save(self) -> None: "created_at": self.created_at, "updated_at": self.updated_at, "error": self.error, + "inputs": self.inputs, } - self._atomic_write_json(runs_dir / "state.json", state_data) - self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs}) + if self.execution is not None: + state_data["execution"] = self.execution + try: + self._atomic_write_json(runs_dir / "state.json", state_data) + self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs}) + except BaseException as exc: + from ._execution import CheckpointError + + self._checkpoint_failed = True + raise CheckpointError(str(exc)) from exc @staticmethod def _atomic_write_json(path: Path, data: dict[str, Any]) -> None: @@ -892,9 +914,14 @@ def load(cls, run_id: str, project_root: Path) -> RunState: state.created_at = state_data.get("created_at", "") state.updated_at = state_data.get("updated_at", "") state.error = state_data.get("error") + state.execution = state_data.get("execution") + if "execution" in state_data: + from ._execution import validate_execution + + validate_execution(state.execution) inputs_path = runs_dir / "inputs.json" - if inputs_path.exists(): + if "inputs" not in state_data and inputs_path.exists(): with open(inputs_path, encoding="utf-8") as f: inputs_data = json.load(f) if not isinstance(inputs_data, dict): @@ -908,6 +935,11 @@ def load(cls, run_id: str, project_root: Path) -> RunState: ) state.inputs = inputs + if "inputs" in state_data: + if not isinstance(state_data["inputs"], dict): + raise ValueError("Invalid run inputs: 'inputs' must be a JSON object") + state.inputs = state_data["inputs"] + return state def append_log(self, entry: dict[str, Any]) -> None: @@ -1089,6 +1121,8 @@ def execute( state.save() return state except Exception as exc: + if state._checkpoint_failed: + raise state.status = RunStatus.FAILED state.error = str(exc) state.append_log({"event": "workflow_failed", "error": str(exc)}) @@ -1106,7 +1140,7 @@ def resume( run_id: str, inputs: dict[str, Any] | None = None, ) -> RunState: - """Resume a paused or failed workflow run. + """Resume a paused/failed run or a tree-backed run interrupted by a crash. When ``inputs`` is provided, the values are merged over the run's persisted inputs and re-resolved through the same typed validation @@ -1115,7 +1149,9 @@ def resume( empty/``None`` ``inputs`` leaves the run's inputs unchanged. """ state = RunState.load(run_id, self.project_root) - if state.status not in (RunStatus.PAUSED, RunStatus.FAILED): + if state.status not in (RunStatus.PAUSED, RunStatus.FAILED) and not ( + state.status == RunStatus.RUNNING and state.execution is not None + ): msg = f"Cannot resume run {run_id!r} with status {state.status.value!r}." raise ValueError(msg) @@ -1144,6 +1180,12 @@ def resume( ) raise ValueError(msg) + if state.execution is not None: + persisted_steps = yaml.safe_load(state.execution["sequence"]["source"]) + offset = state.execution.get("offset", 0) + if persisted_steps != definition.steps[offset:]: + raise ValueError("Invalid execution state: root sequence differs from workflow snapshot") + dispatch_default_errors = _dispatch_default_errors(definition) if dispatch_default_errors: raise ValueError(" ".join(dispatch_default_errors)) @@ -1182,6 +1224,7 @@ def resume( self._execute_steps( remaining_steps, context, state, STEP_REGISTRY, step_offset=step_offset, + rebind=bool(inputs), ) except KeyboardInterrupt: state.status = RunStatus.PAUSED @@ -1189,6 +1232,8 @@ def resume( state.save() return state except Exception as exc: + if state._checkpoint_failed: + raise state.status = RunStatus.FAILED state.error = str(exc) state.append_log({"event": "resume_failed", "error": str(exc)}) @@ -1225,240 +1270,26 @@ def _execute_steps( registry: dict[str, Any], *, step_offset: int = 0, + rebind: bool = False, ) -> None: - """Execute a list of steps sequentially.""" - for i, step_config in enumerate(steps): - step_id = step_config.get("id", f"step-{i}") - step_type = step_config.get("type", "command") - - state.current_step_id = step_id - if step_offset >= 0: - state.current_step_index = step_offset + i - state.save() - - state.append_log( - {"event": "step_started", "step_id": step_id, "type": step_type} - ) - - # Log progress — use the engine's on_step_start callback if set, - # otherwise stay silent (library-safe default). - label = step_config.get("command", "") or step_type - if self.on_step_start is not None: - with self._callback_lock: - self.on_step_start(step_id, label) - - step_impl = registry.get(step_type) - if not step_impl: - state.status = RunStatus.FAILED - state.error = f"Unknown step type: {step_type!r}" - state.append_log( - { - "event": "step_failed", - "step_id": step_id, - "error": f"Unknown step type: {step_type!r}", - } - ) - state.save() - return - - result: StepResult = step_impl.execute(step_config, context) - - # Record step results — prefer resolved values from step output - step_data = { - "type": step_type, - "integration": result.output.get("integration") - or step_config.get("integration") - or context.default_integration, - "model": result.output.get("model") - or step_config.get("model") - or context.default_model, - "options": result.output.get("options") - or step_config.get("options", {}), - "input": result.output.get("input") - or step_config.get("input", {}), - "output": result.output, - "status": result.status.value, - "error": result.error, - } - if step_type == "command" and "integration_args" in result.output: - step_data["integration_args"] = result.output["integration_args"] - step_data["integration_options"] = result.output[ - "integration_options" - ] - self._record_result(context, state, step_id, step_data) - - state.append_log( - { - "event": "step_completed", - "step_id": step_id, - "status": result.status.value, - } - ) - - # Handle gate pauses - if result.status == StepStatus.PAUSED: - state.status = RunStatus.PAUSED - state.save() - return - - # Handle failures - if result.status == StepStatus.FAILED: - # Gate abort (output.aborted) maps to ABORTED status. - # Aborts are deliberate operator decisions, so - # `continue_on_error` does NOT override them — that flag - # is for transient/expected step failures only. - if result.output.get("aborted"): - state.status = RunStatus.ABORTED - state.error = result.error - state.append_log( - { - "event": "workflow_aborted", - "step_id": step_id, - } - ) - state.save() - return - - # `continue_on_error: true` lets the pipeline route - # around the failure instead of halting. The step - # result (including exit_code, stderr, status) is - # still recorded so a downstream `if` or `switch` - # can branch on it (or a `gate` can surface it to the - # operator via message interpolation). Log a single, - # unambiguous event per failure resolution — either - # the run continued past it, or it halted. - # - # Use identity comparison (`is True`) rather than - # truthiness so that only a literal boolean enables - # the behaviour, even if validation was skipped. - # Validation rejects non-bool values at parse time, - # but `WorkflowEngine.execute()` does not auto-validate - # (see `WorkflowEngine.load_workflow`, whose docstring - # explicitly notes "not yet validated; call - # `validate_workflow()` or `engine.validate()` - # separately"), so a caller passing an unvalidated - # definition could otherwise see truthy non-bool - # values like the string `"true"` silently change - # run semantics. - if step_config.get("continue_on_error") is True: - state.append_log( - { - "event": "step_continue_on_error", - "step_id": step_id, - "error": result.error, - } - ) - state.save() - continue - - state.status = RunStatus.FAILED - state.error = result.error - state.append_log( - { - "event": "step_failed", - "step_id": step_id, - "error": result.error, - } - ) - state.save() - return - - # Execute nested steps (from control flow) - # NOTE: Nested steps run with step_offset=-1 so they don't - # update current_step_index. If a nested step pauses, - # resume will re-run the parent step and its nested body. - # A step-path stack for exact nested resume is a future - # enhancement. - if result.next_steps: - self._execute_steps( - result.next_steps, context, state, registry, - step_offset=-1, - ) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - - # Loop iteration: while/do-while re-evaluate after body - if step_type in ("while", "do-while"): - from .expressions import evaluate_condition - - max_iters = step_config.get("max_iterations") - # A bool is an int in Python (isinstance(True, int) is True - # and True == 1), so a bool max_iterations would slip past - # the int check and cap the loop at range(0)==1 iteration - # instead of the default. Exclude bools, mirroring the - # while/do-while validators and the continue_on_error guard. - if ( - isinstance(max_iters, bool) - or not isinstance(max_iters, int) - or max_iters < 1 - ): - max_iters = 10 - condition = step_config.get("condition", False) - for _loop_iter in range(max_iters - 1): - if not evaluate_condition(condition, context): - break - # Namespace nested step IDs per iteration - # so logs and state keys are unique. - # Execute one step at a time and alias each - # result back to the unprefixed key so that - # later steps in the same body and the loop - # condition see the latest values. - for ns_idx, ns in enumerate(result.next_steps): - ns_copy = dict(ns) - orig = ns_copy.get("id") - base_id = orig or f"step-{ns_idx}" - ns_copy["id"] = f"{step_id}:{base_id}:{_loop_iter + 1}" - self._execute_steps( - [ns_copy], context, state, registry, - step_offset=-1, - ) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - if orig and ns_copy["id"] in context.steps: - self._record_result( - context, state, orig, - context.steps[ns_copy["id"]], - ) - - # Fan-out: execute the nested step template once per item. Honors - # max_concurrency — <=1 runs sequentially (default, historical - # behavior); >1 runs up to that many items concurrently. Either way - # results are assembled in item order under the - # parentId:templateId:index id grammar. - if step_type == "fan-out": - items = result.output.get("items", []) - template = result.output.get("step_template", {}) - if template and items: - fan_out_results = self._run_fan_out( - items, template, step_id, context, state, registry, - result.output.get("max_concurrency", 1), - ) - context.item = None - # Preserve original output and add collected results - fan_out_output = dict(result.output) - fan_out_output["results"] = fan_out_results - # set_step_output updates the recorded dict under the run lock; - # context.steps[step_id] is that same object, so it reflects the - # change too — no separate (unlocked) context mutation needed. - state.set_step_output(step_id, fan_out_output) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - else: - # Empty items or no template — normalize output - result.output["results"] = [] - state.set_step_output(step_id, result.output) + """Execute or resume the persisted tree (legacy indices adapt once).""" + from ._execution import Execution, active_step, sequence + from copy import deepcopy + + if state.execution is None: + state.execution = {"version": 1, "sequence": sequence(steps)} + state.execution["offset"] = max(0, step_offset) + state.execution["initial"] = deepcopy(context.steps) + context.steps = deepcopy(state.execution.get("initial", {})) + state.save() + tree = state.execution["sequence"] + executor = Execution(self, state, registry, rebind=rebind) + outcome, error = executor.run(tree, context, (state.workflow_id,), root=True) + active = active_step(state.execution) + if active is not None: + state.current_step_id = active[0][-1] + state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) + state.error = error def _run_fan_out( self, @@ -1470,184 +1301,24 @@ def _run_fan_out( registry: dict[str, Any], max_concurrency: Any, ) -> list[Any]: - """Run a fan-out template once per item; return per-item outputs in item order. - - ``max_concurrency`` <= 1 (the default) runs items sequentially, identical - to the historical fan-out behavior. ``max_concurrency`` > 1 runs items on a - bounded thread pool using a sliding submission window of that size: at most - that many items are ever in flight, and no new item is launched once the run - has reached a halting status, so a halt cannot keep starting queued work. - - Results are always returned in item order (never completion order). On a - halt (PAUSED/FAILED/ABORTED) the returned prefix is the items up to and - including the first item *in item order* whose own execution halted the run - — identical to the sequential path. Later items that have not yet started - are cancelled; any already running are allowed to finish but their outputs - are ignored. Halt is attributed per item from that item's recorded result - (not the shared run status, which a concurrently-running later item may have - already flipped), so the prefix never drops the actual halting item. - - ``max_concurrency`` is coerced with ``int()``; a value that cannot be - coerced (``None``, a non-numeric string, ``.inf``/``.nan``, …) or that - coerces to <= 1 runs sequentially, while a numeric string like ``"4"`` or - a float like ``4.0`` is honored. - """ + """Compatibility adapter to the unified fan-out executor.""" + from ._execution import Execution, sequence + if not items: return [] - - halting = (RunStatus.PAUSED, RunStatus.FAILED, RunStatus.ABORTED) - try: - workers = max(1, int(max_concurrency)) - except (TypeError, ValueError, OverflowError): - # OverflowError: int(float("inf")) — a YAML ``max_concurrency: .inf`` - # would otherwise crash the whole run instead of falling back. - workers = 1 - # Never spin up more workers than there is work — bounds a user-controlled - # max_concurrency from over-allocating threads. - workers = min(workers, len(items)) - - base_id = template.get("id", "item") - - def item_id(idx: int) -> str: - # Per-item ID grammar: parentId:templateId:index. - return f"{step_id}:{base_id}:{idx}" - - def run_item(idx: int, item_ctx: StepContext) -> Any: - item_step = dict(template) - item_step["id"] = item_id(idx) - self._execute_steps( - [item_step], item_ctx, state, registry, step_offset=-1, - ) - # Read back through the context that was actually executed against, - # not the outer closure — clearer and robust if StepContext copying - # ever stops sharing the steps dict by reference. - return item_ctx.steps.get(item_step["id"], {}).get("output", {}) - - # Sequential path — identical to the historical behavior. - if workers <= 1: - results: list[Any] = [] - previous_item = context.item - previous_inside_fan_out = context.inside_fan_out - context.inside_fan_out = True - try: - for item_idx, item_val in enumerate(items): - context.item = item_val - results.append(run_item(item_idx, context)) - if state.status in halting: - break - finally: - context.item = previous_item - context.inside_fan_out = previous_inside_fan_out - return results - - # Concurrent path — bounded sliding window; results assembled in item order. - n = len(items) - slots: list[Any] = [None] * n - - def run_isolated(idx: int) -> Any: - # Each item runs against its own context copy so context.item is not - # clobbered across threads; the shared steps dict is written only on the - # disjoint parentId:templateId:index key (GIL-safe on distinct keys). - return run_item( - idx, - dataclasses.replace( - context, - item=items[idx], - inside_fan_out=True, - ), - ) - - def item_halt_status(idx: int) -> RunStatus | None: - # If THIS item's own execution halted the run, return the resulting run - # status; else None. Decided from the item's own recorded result, not - # the shared run status, so a later item's concurrent halt is never - # misattributed here. Mirrors the sequential mapping: PAUSED -> PAUSED; - # FAILED -> ABORTED when aborted, else FAILED, unless continue_on_error - # routes around it. - rec = context.steps.get(item_id(idx)) - if rec is None: - # Ran but recorded nothing — only when the item failed before - # record_step_result (e.g. an unknown step type returns early). - # Every item runs the same template, so the shared run status is - # this item's own outcome; attribute the halt to it. - return state.status if state.status in halting else None - status = rec.get("status") - if status == StepStatus.PAUSED.value: - return RunStatus.PAUSED - if status == StepStatus.FAILED.value: - out = rec.get("output") or {} - if out.get("aborted"): - return RunStatus.ABORTED - if template.get("continue_on_error") is not True: - return RunStatus.FAILED - return None - - # (halting item index, its run status) once a halt is attributed. - halt: tuple[int, RunStatus] | None = None - collected = 0 - with ThreadPoolExecutor(max_workers=workers) as pool: - futures: dict[int, Future] = {} - next_submit = 0 - for idx in range(n): - # Refill the window: keep <= workers in flight, and stop launching - # new items once the run is halting so a halt cannot keep starting - # queued work. Already-submitted futures are still collected in - # item order below. - while ( - next_submit < n - and len(futures) < workers - and state.status not in halting - ): - futures[next_submit] = pool.submit(run_isolated, next_submit) - next_submit += 1 - - fut = futures.pop(idx, None) - if fut is None: - # Safety net: the window submits indices in order and the loop - # breaks at the first halting item, so every collected index has - # an in-flight future. Stop cleanly rather than raise if a future - # change ever breaks that invariant. - break - try: - slots[idx] = fut.result() - except Exception: - # A genuine exception escaping a step (not a normal step - # FAILED, which sets state.status) must not be masked: cancel - # outstanding work and re-raise — with a bare ``raise`` so the - # original traceback is preserved — so the engine marks the run - # failed instead of reporting a vacuous completion. The pool's - # __exit__ still joins any already-running workers. - for other in futures.values(): - other.cancel() - raise - collected = idx + 1 - halt_status = item_halt_status(idx) - if halt_status is not None: - # First halting item in item order: include it (slots[idx] is - # already set), record its status, and cancel everything pending. - halt = (idx, halt_status) - for other in futures.values(): - other.cancel() - break - - if halt is not None: - halted_at, halted_status = halt - # A later in-flight item may have overwritten state.status before the - # pool joined; restore the halting item's own outcome so the final run - # status matches the sequential semantics. - state.status = halted_status - # Restore the halting item's error so it matches the terminal - # status — a concurrent item may have overwritten state.error - # before the pool joined. Assign unconditionally when a record - # exists (even when the halting item's own error is falsy) so a - # third-party step returning FAILED with no message never inherits - # an unrelated concurrent item's error; this mirrors the sequential - # path, which sets state.error = result.error verbatim. - halt_rec = context.steps.get(item_id(halted_at)) - if isinstance(halt_rec, dict): - state.error = halt_rec.get("error") - return slots[: halted_at + 1] - return slots[:collected] + node = { + "phase": "children", + "result": {"output": {"items": items, "step_template": template, + "max_concurrency": max_concurrency}}, + "children": [sequence([template]) for _ in items], + } + outcome, error, results = Execution(self, state, registry).fan_out( + {"id": step_id, "type": "fan-out"}, node, context, + (state.workflow_id,), (), step_id, + ) + state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) + state.error = error + return results def _resolve_inputs( self, diff --git a/src/specify_cli/workflows/step/gate/__init__.py b/src/specify_cli/workflows/step/gate/__init__.py index 5aac060c0f..111ca960ed 100644 --- a/src/specify_cli/workflows/step/gate/__init__.py +++ b/src/specify_cli/workflows/step/gate/__init__.py @@ -40,6 +40,8 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: message = config.get("message", "Review required.") if isinstance(message, str) and "{{" in message: message = evaluate_expression(message, context) + if message is not None: + message = str(message) options = config.get("options", ["approve", "reject"]) on_reject = config.get("on_reject", "abort") diff --git a/src/specify_cli/workflows/step/workflow/__init__.py b/src/specify_cli/workflows/step/workflow/__init__.py new file mode 100644 index 0000000000..0c8f1df328 --- /dev/null +++ b/src/specify_cli/workflows/step/workflow/__init__.py @@ -0,0 +1,17 @@ +"""Installed workflow calls are scoped execution handled by the engine.""" + +from ...base import StepBase, StepResult, StepStatus +from ...composition import validate_call + + +class WorkflowStep(StepBase): + type_key = "workflow" + + def validate(self, config): + return [*super().validate(config), *validate_call(config)] + + def execute(self, config, context): + return StepResult( + status=StepStatus.FAILED, + error="Workflow calls require the workflow engine", + ) diff --git a/tests/specify_cli/workflows/test_command_status.py b/tests/specify_cli/workflows/test_command_status.py index 0a7b50ac13..16f6a01352 100644 --- a/tests/specify_cli/workflows/test_command_status.py +++ b/tests/specify_cli/workflows/test_command_status.py @@ -95,6 +95,38 @@ def test_status_json_single_and_list(self, project_dir): ) assert any(r["run_id"] == rid for r in listing["runs"]) + def test_composed_gate_status_and_resume(self, project_dir): + from specify_cli.workflows.catalog import WorkflowRegistry + + child_dir = project_dir / ".specify" / "workflows" / "child" + child_dir.mkdir(parents=True) + (child_dir / "workflow.yml").write_text( + "workflow: {id: child, name: Child}\n" + "inputs:\n verdict: {type: string, default: ''}\n" + "steps:\n - {id: review, type: gate, message: Review, verdict_input: verdict}\n", + encoding="utf-8", + ) + WorkflowRegistry(project_dir).add("child", {"enabled": True}) + root = self._write_wf( + project_dir, + "workflow: {id: parent, name: Parent}\n" + "inputs:\n verdict: {type: string, default: ''}\n" + "steps:\n - id: call\n type: workflow\n workflow: child\n" + " input: {verdict: '{{ inputs.verdict }}'}\n", + "parent", + ) + run = json.loads(self._invoke(project_dir, ["workflow", "run", str(root), "--json"]).stdout) + assert run["status"] == "paused", run + status = json.loads(self._invoke(project_dir, ["workflow", "status", run["run_id"], "--json"]).stdout) + assert status["gate"] == run["gate"] + assert status["gate"]["scope_path"] == ["call"] + assert status["workflow_scopes"] == [{"scope_path": ["call"], "workflow_id": "child", "status": "paused"}] + human = self._invoke(project_dir, ["workflow", "status", run["run_id"]]) + assert "child: paused" in human.stdout + resumed = self._invoke(project_dir, ["workflow", "resume", run["run_id"], "--input", "verdict=approve", "--json"]) + assert resumed.exit_code == 0 + assert json.loads(resumed.stdout)["status"] == "completed" + class TestWorkflowCliAlignment: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py new file mode 100644 index 0000000000..ae1bc88859 --- /dev/null +++ b/tests/workflows/test_composition_execution.py @@ -0,0 +1,670 @@ +"""Behavioral contracts for scoped calls and durable execution occurrences.""" + +from collections import Counter +from datetime import date +import json +import threading + +import pytest +import yaml + +from specify_cli.workflows import STEP_REGISTRY +from specify_cli.workflows.base import RunStatus, StepBase, StepResult, StepStatus +from specify_cli.workflows.engine import RunState, WorkflowDefinition, WorkflowEngine + + +def definition(name, steps, **fields): + return WorkflowDefinition( + {"workflow": {"id": name, "name": name}, "steps": steps, **fields} + ) + + +def install(root, child, enabled=True): + from specify_cli.workflows.catalog import WorkflowRegistry + + directory = root / ".specify" / "workflows" / child.id + directory.mkdir(parents=True, exist_ok=True) + (directory / "workflow.yml").write_text( + yaml.safe_dump(child.data), encoding="utf-8" + ) + registry = WorkflowRegistry(root) + registry.add(child.id, {"version": "1.0.0", "enabled": enabled}) + return directory + + +def call(target="child", **extra): + return {"id": "call", "type": "workflow", "workflow": target, **extra} + + +@pytest.fixture +def probe(monkeypatch): + counts = Counter() + + class Probe(StepBase): + type_key = "probe" + + def execute(self, config, context): + from specify_cli.workflows.expressions import evaluate_expression + + counts[config["id"]] += 1 + value = evaluate_expression(config.get("value"), context) + status = config.get("status", "completed") + if config.get("await") and not context.inputs.get("approve"): + status = "paused" + return StepResult( + StepStatus(status), output={"value": value, **config.get("output", {})} + ) + + monkeypatch.setitem(STEP_REGISTRY, "probe", Probe()) + return counts + + +def test_declared_output_and_scope_isolation(tmp_path, probe): + child = definition( + "child", + [ + {"id": "inspect", "type": "probe", "value": "{{ inputs.value }}"}, + { + "id": "private", + "type": "probe", + "value": "{{ steps.parent.output.value }}", + }, + ], + inputs={"value": {"type": "string"}}, + outputs={ + "value": {"value": "{{ steps.inspect.output.value }}"}, + "hidden": {"value": "{{ steps.private.output.value }}"}, + }, + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "parent", "type": "probe", "value": "secret"}, + call(input={"value": "mapped"}), + { + "id": "consume", + "type": "probe", + "value": "{{ steps.call.output.value }}", + }, + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"] == { + "workflow": "child", + "status": "completed", + "value": "mapped", + "hidden": None, + } + assert state.step_results["consume"]["output"]["value"] == "mapped" + assert "inspect" not in state.step_results + assert len(list((tmp_path / ".specify/workflows/runs").iterdir())) == 1 + + +def test_concurrent_nested_calls_keep_downstream_aliases_local( + tmp_path, monkeypatch, probe +): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "number"}}, + outputs={"value": {"value": "{{ inputs.value }}"}}, + ), + ) + barrier = threading.Barrier(2, timeout=5) + consumed = {} + + class Consume(StepBase): + type_key = "consume" + + def execute(self, config, context): + barrier.wait() + consumed[context.item] = context.steps["call"]["output"]["value"] + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "consume", Consume()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "spread", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": { + "id": "branch", + "type": "if", + "condition": True, + "then": [ + call(input={"value": "{{ item }}"}), + {"id": "consume", "type": "consume"}, + ], + }, + } + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert consumed == {1: 1, 2: 2} + items = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0][ + "children" + ] + results = [ + item["nodes"][0]["children"][0]["nodes"][0]["result"]["output"]["value"] + for item in items + ] + assert results == [1, 2] + assert "call" not in state.step_results + + +def test_nested_resume_freezes_branch_binding_and_completed_prefix(tmp_path, probe): + child = definition( + "child", + [ + {"id": "prepare", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + directory = install(tmp_path, child) + root = definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": "{{ inputs.choose }}", + "then": [call(input={"approve": "{{ inputs.approve }}"})], + "else": [{"id": "wrong", "type": "probe"}], + } + ], + inputs={ + "choose": {"type": "boolean", "default": True}, + "approve": {"type": "boolean", "default": False}, + }, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + (directory / "workflow.yml").write_text("invalid", encoding="utf-8") + state = WorkflowEngine(tmp_path).resume(state.run_id, {"choose": False}) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"prepare": 1, "wait": 3} + + +@pytest.mark.parametrize( + "status,aborted,handled,expected", + [ + ("failed", False, False, "failed"), + ("failed", False, True, "completed"), + ("failed", True, True, "aborted"), + ("paused", False, True, "paused"), + ], +) +def test_call_outcomes(tmp_path, probe, status, aborted, handled, expected): + install( + tmp_path, + definition( + "child", + [ + { + "id": "work", + "type": "probe", + "status": status, + "output": {"aborted": aborted}, + }, + ], + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(continue_on_error=handled), + {"id": "after", "type": "probe"}, + ], + ) + ) + assert state.status.value == expected + assert probe["after"] == (expected == "completed") + assert state.step_results["call"]["output"]["status"] == ( + "aborted" if aborted else status + ) + + +@pytest.mark.parametrize( + "target", [" child", "child\n", "CHILD", "../child", "runs", 123, date(2026, 1, 1)] +) +def test_invalid_targets_are_handled_failures(tmp_path, probe, target): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(target, continue_on_error=True)]) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + RunState.load(state.run_id, tmp_path) + + +@pytest.mark.parametrize( + "mapping", [{"unknown": "x"}, {"value": []}, {"value": date(2026, 1, 1)}, [1]] +) +def test_invalid_inputs_do_not_execute_child(tmp_path, probe, mapping): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string", "required": True}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(input=mapping)]) + ) + assert state.status == RunStatus.FAILED + assert not probe + RunState.load(state.run_id, tmp_path) + + +def test_output_failure_retries_only_finalization(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + original = execution.evaluate_outputs + monkeypatch.setattr( + execution, + "evaluate_outputs", + lambda *_: (_ for _ in ()).throw(ValueError("bad output")), + ) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + assert state.status == RunStatus.FAILED + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.FAILED + monkeypatch.setattr(execution, "evaluate_outputs", original) + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +def test_legacy_resume_adapts_once(tmp_path, probe): + root = definition( + "parent", + [ + {"id": "before", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + del data["execution"] + path.write_text(json.dumps(data)) + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"before": 1, "wait": 3} + + +@pytest.mark.parametrize( + "mutation", + [ + lambda tree: tree.update(version=99), + lambda tree: tree["sequence"].update(nodes=[]), + lambda tree: tree["sequence"]["nodes"][0].update(phase="nonsense"), + ], +) +def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]) + path.write_text(json.dumps(data)) + before = path.read_bytes() + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id) + assert path.read_bytes() == before + + +@pytest.mark.parametrize("failure_after_replace", [False, True]) +def test_checkpoint_failure_never_overwrites_committed_progress( + tmp_path, monkeypatch, probe, failure_after_replace +): + from specify_cli.workflows._execution import CheckpointError + + original = RunState._atomic_write_json + failed = False + + def write(path, data): + nonlocal failed + record = data.get("step_results", {}).get("work") + if path.name == "state.json" and record and not failed: + failed = True + if failure_after_replace: + original(path, data) + raise OSError("checkpoint failure") + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + with pytest.raises(CheckpointError, match="checkpoint failure"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "work", "type": "probe"}]), run_id="fault" + ) + disk = json.loads( + (tmp_path / ".specify/workflows/runs/fault/state.json").read_text() + ) + node = disk["execution"]["sequence"]["nodes"][0] + assert node["phase"] == ("done" if failure_after_replace else "ready") + assert ("work" in disk["step_results"]) is failure_after_replace + state = WorkflowEngine(tmp_path).resume("fault") + assert state.status == RunStatus.COMPLETED + assert probe["work"] == (1 if failure_after_replace else 2) + + +def test_completion_log_failure_does_not_replay_committed_step( + tmp_path, monkeypatch, probe +): + original = RunState.append_log + failed = False + + def log(self, entry): + nonlocal failed + if entry["event"] == "step_completed" and not failed: + failed = True + raise OSError("log failed") + original(self, entry) + + monkeypatch.setattr(RunState, "append_log", log) + with pytest.raises(OSError, match="log failed"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "work", "type": "probe"}]), run_id="log" + ) + state = WorkflowEngine(tmp_path).resume("log") + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +@pytest.mark.parametrize("kind", ["if", "while", "do-while", "fan-out"]) +def test_expansion_resume_preserves_completed_work(tmp_path, probe, kind): + body = [ + {"id": "prepare", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ] + config = {"id": "outer", "type": kind, "condition": True, "max_iterations": 2} + if kind == "if": + config["then"] = body + elif kind == "fan-out": + config.update( + items=[1, 2], + max_concurrency=2, + step={"id": "branch", "type": "if", "condition": True, "then": body}, + ) + else: + config["steps"] = body + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [config], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe["prepare"] == (1 if kind == "if" else 2) + + +@pytest.mark.parametrize("mode", ["unknown", "disabled", "mismatch", "cycle"]) +def test_resolution_failures_are_call_failures(tmp_path, probe, mode): + if mode != "unknown": + child = definition( + "child", + [call("parent") if mode == "cycle" else {"id": "work", "type": "probe"}], + ) + directory = install(tmp_path, child, enabled=mode != "disabled") + if mode == "mismatch": + child.data["workflow"]["id"] = "other" + (directory / "workflow.yml").write_text(yaml.safe_dump(child.data)) + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert not probe + + +def test_diamond_and_depth_limit(tmp_path, probe): + install(tmp_path, definition("leaf", [{"id": "work", "type": "probe"}])) + for name in ("left", "right"): + install(tmp_path, definition(name, [call("leaf")])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {**call("left"), "id": "left"}, + {**call("right"), "id": "right"}, + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 2 + for index in range(1, 18): + install( + tmp_path, + definition( + f"level-{index}", + [call(f"level-{index + 1}")] + if index < 17 + else [{"id": "too-deep", "type": "probe"}], + ), + ) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call("level-1")])) + assert state.status == RunStatus.FAILED + assert "depth 16" in state.error + assert probe["too-deep"] == 0 + + +def test_output_validation_rejects_reserved_and_yaml_native_values(tmp_path, probe): + from specify_cli.workflows.composition import require_json, validate_outputs + + circular = [] + circular.append(circular) + for value in (circular, date(2026, 1, 1), {1: "key"}, float("nan")): + with pytest.raises(ValueError, match="JSON-safe"): + require_json(value) + for outputs in ( + {"status": {"value": "oops"}}, + {"value": {"wrong": 1}}, + {"value": {"value": date(2026, 1, 1)}}, + ): + assert validate_outputs(outputs) + + +def test_nested_gate_reporting_uses_active_occurrence(tmp_path, probe): + from specify_cli.workflows._commands import _workflow_run_payload + + install( + tmp_path, + definition( + "child", + [{"id": "review", "type": "gate", "message": "Approve", "mode": "manual"}], + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [call()], + } + ], + ) + ) + assert state.status == RunStatus.PAUSED + payload = _workflow_run_payload(RunState.load(state.run_id, tmp_path)) + assert payload["gate"]["step_id"] == "review" + assert payload["gate"]["scope_path"] == ["route", "call"] + assert payload["workflow_scopes"][0]["status"] == "paused" + + +def test_rebind_failure_has_one_failed_caller_outcome(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean"}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"}, continue_on_error=True)], + inputs={"approve": {"type": "string", "default": "false"}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "invalid"}) + assert state.status == RunStatus.COMPLETED + node = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0] + assert node["phase"] == "done" + assert node["result"]["output"]["status"] == "failed" + from specify_cli.workflows._execution import active_step + + assert active_step(state.execution) is None + + +@pytest.mark.parametrize( + "status,handled", [("failed", False), ("paused", False), ("failed", True)] +) +def test_unsuccessful_expansion_never_executes_children( + tmp_path, monkeypatch, probe, status, handled +): + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + return StepResult( + StepStatus(status), next_steps=[{"id": "wrong", "type": "probe"}] + ) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "expand", + "type": "expand", + "continue_on_error": handled, + } + ], + ) + ) + assert state.status.value == ("completed" if handled else status) + assert not probe + RunState.load(state.run_id, tmp_path) + + +def test_custom_expansion_is_frozen_across_resume(tmp_path, monkeypatch, probe): + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + probe["expand"] += 1 + return StepResult( + StepStatus.COMPLETED, + next_steps=[ + {"id": f"prefix-{probe['expand']}", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + root = definition( + "parent", + [{"id": "expand", "type": "expand"}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"expand": 1, "prefix-1": 1, "wait": 2} + + +def test_native_yaml_definition_and_long_id_roundtrip(tmp_path, probe): + target = "a" * 240 + child = definition( + target, [{"id": "review", "type": "gate", "message": date(2026, 1, 1)}] + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call(target)])) + assert state.status == RunStatus.PAUSED + saved = RunState.load(state.run_id, tmp_path) + binding = saved.execution["sequence"]["nodes"][0]["binding"] + assert yaml.safe_load(binding["definition"])["steps"][0]["message"] == date( + 2026, 1, 1 + ) + assert WorkflowEngine(tmp_path).resume(state.run_id).status == RunStatus.PAUSED + + +def test_exact_depth_limit_is_allowed(tmp_path, probe): + for index in range(1, 17): + install( + tmp_path, + definition( + f"level-{index}", + [call(f"level-{index + 1}")] + if index < 16 + else [{"id": "work", "type": "probe"}], + ), + ) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call("level-1")])) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +def test_aborted_fanout_sibling_is_never_restarted(tmp_path, monkeypatch): + barrier = threading.Barrier(2, timeout=5) + counts = Counter() + + class Mixed(StepBase): + type_key = "mixed" + + def execute(self, config, context): + counts[context.item] += 1 + if not context.is_resume: + barrier.wait() + if context.item == 0: + return StepResult( + StepStatus.COMPLETED if context.is_resume else StepStatus.PAUSED + ) + return StepResult(StepStatus.FAILED, output={"aborted": True}) + + monkeypatch.setitem(STEP_REGISTRY, "mixed", Mixed()) + root = definition( + "parent", + [ + { + "id": "spread", + "type": "fan-out", + "items": [0, 1], + "max_concurrency": 2, + "step": {"id": "mixed", "type": "mixed"}, + } + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.ABORTED + assert counts == {0: 2, 1: 1} diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index 1b06dc2e66..a9ee91c00d 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -72,14 +72,16 @@ flowchart LR When a `gate` step pauses execution, the engine persists `current_step_index` and all accumulated `step_results`. On `specify workflow resume `, the engine restores the context and continues from the paused step. -> **Note:** Resume tracking is at the top-level step index only. If a -> nested step (inside `if`/`switch`/`while`) pauses, resume re-runs -> the parent control-flow step and its nested body. A nested step-path -> stack for exact resume is a planned enhancement. +New runs use a versioned execution tree. Each occurrence owns its result, +selected child sequences, and optional workflow binding. The same executor +traverses that tree on initial execution and resume, restoring local aliases +from completed results. Fan-out items have separate contexts. Legacy runs enter +through their top-level index once. Inputs and tree transitions share one atomic +state checkpoint; the inputs file is a compatibility mirror. ## Step Types -The engine ships with 12 built-in step types, each in its own subpackage under `src/specify_cli/workflows/step/`: +The engine ships with 13 built-in step types, each in its own subpackage under `src/specify_cli/workflows/step/`: | Type Key | Class | Purpose | Returns `next_steps`? | |----------|-------|---------|-----------------------| @@ -95,6 +97,7 @@ The engine ships with 12 built-in step types, each in its own subpackage under ` | `do-while` | `DoWhileStep` | Loop, always runs body at least once | Yes (always) | | `fan-out` | `FanOutStep` | Dispatch per item over a collection | No (engine expands) | | `fan-in` | `FanInStep` | Aggregate results from fan-out | No | +| `workflow` | `WorkflowStep` | Execute an installed workflow in a private scope | No (engine enters scope) | ## Step Registry diff --git a/workflows/PUBLISHING.md b/workflows/PUBLISHING.md index 4e26fdfb25..5b7850cbe0 100644 --- a/workflows/PUBLISHING.md +++ b/workflows/PUBLISHING.md @@ -90,7 +90,7 @@ steps: - ✅ `version` follows semantic versioning (X.Y.Z) - ✅ `description` is concise - ✅ All step IDs are unique -- ✅ Step types are valid: `command`, `prompt`, `shell`, `init`, `slot`, `gate`, `if`, `switch`, `while`, `do-while`, `fan-out`, `fan-in` +- ✅ Step types are valid: `command`, `prompt`, `shell`, `init`, `slot`, `gate`, `if`, `switch`, `while`, `do-while`, `fan-out`, `fan-in`, `workflow` - ✅ Required fields present per step type (e.g., `condition` for `if`, `expression` for `switch`) - ✅ Input types are valid: `string`, `number`, `boolean` - ✅ Step IDs do not contain `:` (reserved for engine-generated nested IDs like `parentId:childId`) diff --git a/workflows/README.md b/workflows/README.md index da045bfdf9..28ac151288 100644 --- a/workflows/README.md +++ b/workflows/README.md @@ -85,7 +85,10 @@ The bundled `speckit` workflow only declares `spec` (and optional ## Step Types -Workflows support 12 built-in step types: +Workflows support 13 built-in step types, including `workflow` for calling an +installed workflow with private inputs and declared outputs. See +[workflow composition and resume](../docs/reference/workflows.md#workflow-composition) +for the scope and execution identity contracts. ### Command Steps (default) From 4dde245711dba38f46bce2abf3c62ba172e1ac32 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:06:27 +0200 Subject: [PATCH 02/28] fix(workflows): complete composition cleanup phase one Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- docs/reference/workflows.md | 4 +- src/specify_cli/workflows/_commands.py | 2 +- src/specify_cli/workflows/_execution.py | 27 ++- src/specify_cli/workflows/command_resume.py | 2 +- src/specify_cli/workflows/engine.py | 77 +------ .../workflows/step/fan_out/__init__.py | 12 +- .../workflows/step/gate/__init__.py | 2 + tests/test_workflows.py | 213 ++++++++++-------- tests/workflows/test_composition_execution.py | 125 +++++++++- 9 files changed, 277 insertions(+), 187 deletions(-) diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index c27ece7342..939791f5ab 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -601,9 +601,7 @@ commands. Snapshots are stored as YAML strings inside the private JSON execution tree, preserving YAML scalar types. Inputs and results remain JSON values. Legacy runs without a tree enter through their saved top-level index, then use tree-backed -resume. A tree-backed run left `running` by a crashed process can also be resumed; -only one process may execute or resume a run at a time. A side effect completed -before its checkpoint may execute again after a crash. +resume. Resume is available only for runs in the `paused` or `failed` state. Run/resume/status JSON includes `workflow_scopes` summaries when calls exist and reports the active nested gate with its `scope_path`. diff --git a/src/specify_cli/workflows/_commands.py b/src/specify_cli/workflows/_commands.py index 7d5757504c..e12d8d06a3 100644 --- a/src/specify_cli/workflows/_commands.py +++ b/src/specify_cli/workflows/_commands.py @@ -999,7 +999,7 @@ def _gate_outcome(state: Any) -> dict[str, Any] | None: if not isinstance(step, dict) or not _is_gate_step(step): return None output = step.get("output") or {} - # `message`, `options`, and `choice` may be non-string YAML literals in an + # `message`, `options`, and `choice` may be non-string YAML literals in # legacy or synthetic records, so # normalise all three for a stable JSON schema: message → str, options → # list[str] | None, choice → str | None (None means no decision yet). diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 7f540af0b9..cc16871c5f 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -283,11 +283,22 @@ def step(self, config, node, context, ancestry, path, public_name): with self.engine._callback_lock: self.engine.on_step_start(name, config.get("command", "") or kind) impl = self.registry.get(kind) - result = ( - impl.execute(config, context) - if impl - else StepResult(StepStatus.FAILED, error=f"Unknown step type: {kind!r}") - ) + if impl is None: + # A missing implementation is a runtime configuration error, not + # a step failure that a workflow may explicitly recover from. + result = StepResult( + StepStatus.FAILED, error=f"Unknown step type: {kind!r}" + ) + return self.finish( + {**config, "continue_on_error": False}, + node, + result, + context, + ancestry, + path, + public_name, + ) + result = impl.execute(config, context) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: return self.finish( config, node, result, context, ancestry, path, public_name @@ -515,6 +526,12 @@ def workflow(self, config, node, context, ancestry, path, public_name): output={"workflow": target, "status": "failed", "error": str(exc)}, error=str(exc), ) + # An unavailable child implementation is terminal at every workflow + # boundary, matching direct execution of the same step. + if isinstance(result.error, str) and result.error.startswith( + "Unknown step type: " + ): + config = {**config, "continue_on_error": False} return self.finish(config, node, result, context, ancestry, path, public_name) def fan_out(self, config, node, context, ancestry, path, public_name): diff --git a/src/specify_cli/workflows/command_resume.py b/src/specify_cli/workflows/command_resume.py index 9a50a6b50c..3be24f9582 100644 --- a/src/specify_cli/workflows/command_resume.py +++ b/src/specify_cli/workflows/command_resume.py @@ -18,7 +18,7 @@ def workflow_resume( help="Emit the resume outcome as a single JSON object instead of formatted text.", ), ): - """Resume a paused, failed, or crash-interrupted workflow run.""" + """Resume a paused or failed workflow run.""" from . import load_custom_steps from .engine import RunState, WorkflowEngine diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index d947f355c5..7fac4f7213 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -720,32 +720,6 @@ def __init__( def runs_dir(self) -> Path: return self.project_root / ".specify" / "workflows" / "runs" / self.run_id - def record_step_result(self, step_id: str, data: dict[str, Any]) -> None: - """Record one step's result under the run lock. - - Routing the mutation through the lock keeps it from racing a concurrent - ``save()`` that is iterating ``step_results`` (e.g. during a concurrent - fan-out). For a sequential run this is an uncontended lock. - """ - with self._lock: - from ._execution import CheckpointError - - if self._checkpoint_failed: - raise CheckpointError("A previous checkpoint failed; reload the run") - self.step_results[step_id] = data - - def set_step_output(self, step_id: str, output: Any) -> None: - """Replace an already-recorded step's ``output`` under the run lock. - - Fan-out updates its parent step's output after the items have run; - routing that nested mutation through the lock keeps it from racing a - ``save()`` serializing ``step_results`` — the same invariant - ``record_step_result`` provides for the top-level assignment. - """ - with self._lock: - if step_id in self.step_results: - self.step_results[step_id]["output"] = output - def save(self) -> None: """Persist current state to disk. @@ -1140,7 +1114,7 @@ def resume( run_id: str, inputs: dict[str, Any] | None = None, ) -> RunState: - """Resume a paused/failed run or a tree-backed run interrupted by a crash. + """Resume a paused or failed workflow run. When ``inputs`` is provided, the values are merged over the run's persisted inputs and re-resolved through the same typed validation @@ -1149,9 +1123,7 @@ def resume( empty/``None`` ``inputs`` leaves the run's inputs unchanged. """ state = RunState.load(run_id, self.project_root) - if state.status not in (RunStatus.PAUSED, RunStatus.FAILED) and not ( - state.status == RunStatus.RUNNING and state.execution is not None - ): + if state.status not in (RunStatus.PAUSED, RunStatus.FAILED): msg = f"Cannot resume run {run_id!r} with status {state.status.value!r}." raise ValueError(msg) @@ -1246,22 +1218,6 @@ def resume( state.save() return state - @staticmethod - def _record_result( - context: StepContext, state: RunState, step_id: str, data: dict[str, Any] - ) -> None: - """Record a step result into both the live context and persistent state. - - ``record_step_result`` writes ``state.step_results`` under the run lock. - On a resume run ``context.steps`` *is* that same dict, so that locked - write is the only one needed; mirror into ``context.steps`` separately - only when it is a distinct object (a fresh run), to avoid an unlocked - mutation of the shared dict that could race a concurrent ``save()``. - """ - if context.steps is not state.step_results: - context.steps[step_id] = data - state.record_step_result(step_id, data) - def _execute_steps( self, steps: list[dict[str, Any]], @@ -1291,35 +1247,6 @@ def _execute_steps( state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) state.error = error - def _run_fan_out( - self, - items: list[Any], - template: dict[str, Any], - step_id: str, - context: StepContext, - state: RunState, - registry: dict[str, Any], - max_concurrency: Any, - ) -> list[Any]: - """Compatibility adapter to the unified fan-out executor.""" - from ._execution import Execution, sequence - - if not items: - return [] - node = { - "phase": "children", - "result": {"output": {"items": items, "step_template": template, - "max_concurrency": max_concurrency}}, - "children": [sequence([template]) for _ in items], - } - outcome, error, results = Execution(self, state, registry).fan_out( - {"id": step_id, "type": "fan-out"}, node, context, - (state.workflow_id,), (), step_id, - ) - state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) - state.error = error - return results - def _resolve_inputs( self, definition: WorkflowDefinition, diff --git a/src/specify_cli/workflows/step/fan_out/__init__.py b/src/specify_cli/workflows/step/fan_out/__init__.py index d961170c8c..5161377c24 100644 --- a/src/specify_cli/workflows/step/fan_out/__init__.py +++ b/src/specify_cli/workflows/step/fan_out/__init__.py @@ -15,7 +15,7 @@ class FanOutStep(StepBase): setting ``context.item`` for each iteration. ``max_concurrency`` controls parallelism: ``<= 1`` (the default) runs items sequentially, while ``> 1`` runs up to that many items concurrently - on a bounded thread pool (see ``WorkflowEngine._run_fan_out``). + on a bounded thread pool. """ type_key = "fan-out" @@ -28,12 +28,10 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: # The engine does not auto-validate step config (see # ``WorkflowEngine.load_workflow``). On a COMPLETED fan-out it reads the - # ``step_template`` back out and, when it is truthy, calls - # ``template.get("id", ...)`` in ``_run_fan_out``. A truthy non-mapping - # ``step`` (a scalar or list authoring mistake) would crash the whole - # run with AttributeError there — the engine invokes ``execute`` and - # ``_run_fan_out`` with no surrounding try/except. ``validate`` already - # rejects a non-mapping ``step``; fail this step loudly on an + # ``step_template`` back out and calls ``template.get("id", ...)`` while + # expanding items. A truthy non-mapping ``step`` (a scalar or list + # authoring mistake) would crash the whole run with AttributeError there. + # ``validate`` already rejects a non-mapping ``step``; fail this step loudly on # unvalidated run instead, mirroring the ``items`` guard below. An empty # or absent ``step`` defaults to ``{}`` (falsy) and the engine's # ``if template and items`` skips fan-out, so it stays valid here. diff --git a/src/specify_cli/workflows/step/gate/__init__.py b/src/specify_cli/workflows/step/gate/__init__.py index 111ca960ed..4297016d36 100644 --- a/src/specify_cli/workflows/step/gate/__init__.py +++ b/src/specify_cli/workflows/step/gate/__init__.py @@ -40,6 +40,8 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: message = config.get("message", "Review required.") if isinstance(message, str) and "{{" in message: message = evaluate_expression(message, context) + # Step results are persisted as JSON. Keep the display value serializable + # while _compose_prompt() independently accepts synthetic non-string data. if message is not None: message = str(message) diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 597ca7112e..cd10c744bb 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -4146,7 +4146,7 @@ def test_execute_non_dict_step_fails_loudly(self): ``validate`` rejects a non-dict ``step``, but the engine's ``execute()`` does not auto-validate (see ``WorkflowEngine.load_workflow``). On a COMPLETED fan-out the engine reads ``step_template`` back out and, when - it is truthy, calls ``template.get("id", ...)`` in ``_run_fan_out``. A + it is truthy, calls ``template.get("id", ...)`` while expanding items. A truthy non-mapping ``step`` (a scalar or list authoring mistake) raised AttributeError there and took down the whole run. Mirrors the fan-out non-list ``items`` guard and the switch non-dict ``cases`` guard. @@ -4377,24 +4377,16 @@ def test_validate_accepts_mapping_or_absent_output(self): class TestFanOutConcurrency: - """Fan-out honors max_concurrency (WorkflowEngine._run_fan_out).""" + """Fan-out honors max_concurrency through WorkflowEngine.execute().""" @staticmethod - def _build(tmp_path, on_item=None): - """Wire an engine + run state to a probe step that echoes context.item. - - Per-item output is ``{"seen": }`` so order and per-thread item - isolation are checkable. ``on_item(item)`` may run a side effect and - optionally return a StepStatus to override COMPLETED (or raise). - """ - from specify_cli.workflows.base import ( - RunStatus, - StepBase, - StepContext, - StepResult, - StepStatus, - ) - from specify_cli.workflows.engine import RunState, WorkflowEngine + def _run( + tmp_path, monkeypatch, items, max_concurrency, on_item=None, template=None + ): + """Run a public fan-out workflow with a probe that echoes ``context.item``.""" + from specify_cli.workflows import STEP_REGISTRY + from specify_cli.workflows.base import StepBase, StepResult, StepStatus + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine class _ProbeStep(StepBase): type_key = "probe" @@ -4407,35 +4399,41 @@ def execute(self, config, context): status = override return StepResult(status=status, output={"seen": context.item}) - engine = WorkflowEngine(project_root=tmp_path) - context = StepContext() - state = RunState(run_id="r", workflow_id="w", project_root=tmp_path) - state.status = RunStatus.RUNNING - template = {"id": "impl", "type": "probe"} - return engine, context, state, {"probe": _ProbeStep()}, template - - def _run(self, tmp_path, items, max_concurrency, on_item=None): - engine, context, state, registry, template = self._build(tmp_path, on_item) - results = engine._run_fan_out( - items, template, "fan", context, state, registry, max_concurrency + monkeypatch.setitem(STEP_REGISTRY, "probe", _ProbeStep()) + template = template or {"id": "impl", "type": "probe"} + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition( + { + "workflow": {"id": "fan-out-test", "name": "Fan-out Test"}, + "steps": [ + { + "id": "fan", + "type": "fan-out", + "items": items, + "max_concurrency": max_concurrency, + "step": template, + } + ], + } + ) ) - return results, state + return state.step_results["fan"]["output"]["results"], state - def test_sequential_default_preserves_order(self, tmp_path): - results, _ = self._run(tmp_path, list(range(5)), 1) + def test_sequential_default_preserves_order(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(5)), 1) assert results == [{"seen": i} for i in range(5)] - def test_concurrent_runs_all_items_in_item_order(self, tmp_path): - results, _ = self._run(tmp_path, list(range(10)), 4) + def test_concurrent_runs_all_items_in_item_order(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(10)), 4) assert results == [{"seen": i} for i in range(10)] - def test_sequential_and_concurrent_agree(self, tmp_path): + def test_sequential_and_concurrent_agree(self, tmp_path, monkeypatch): items = [{"n": i} for i in range(8)] - seq, _ = self._run(tmp_path, items, 1) - con, _ = self._run(tmp_path, items, 4) + seq, _ = self._run(tmp_path, monkeypatch, items, 1) + con, _ = self._run(tmp_path, monkeypatch, items, 4) assert seq == con == [{"seen": {"n": i}} for i in range(8)] - def test_shuffled_completion_preserves_item_order(self, tmp_path): + def test_shuffled_completion_preserves_item_order(self, tmp_path, monkeypatch): # Determinism keystone: completion order is forced to the exact REVERSE of # item order by an event chain (no sleeps) — item i blocks until item i+1 # has finished, so item 0 completes LAST — yet results must still be in @@ -4455,11 +4453,11 @@ def on_item(item): done[item].set() return None - results, _ = self._run(tmp_path, list(range(n)), n, on_item) + results, _ = self._run(tmp_path, monkeypatch, list(range(n)), n, on_item) assert results == [{"seen": i} for i in range(n)] assert completion == list(reversed(range(n))) - def test_concurrency_is_real(self, tmp_path): + def test_concurrency_is_real(self, tmp_path, monkeypatch): import threading # Deterministic proof of real parallelism (no wall-clock threshold to @@ -4473,30 +4471,35 @@ def on_item(item): barrier.wait() return None - results, _ = self._run(tmp_path, list(range(n)), n, on_item) + results, _ = self._run(tmp_path, monkeypatch, list(range(n)), n, on_item) assert results == [{"seen": i} for i in range(n)] @pytest.mark.parametrize( "bad", [0, -1, None, "abc", 1.0, float("inf"), float("nan")] ) - def test_invalid_max_concurrency_coerces_to_sequential(self, tmp_path, bad): + def test_invalid_max_concurrency_coerces_to_sequential( + self, tmp_path, monkeypatch, bad + ): # float("inf") -> int() raises OverflowError (not TypeError/ValueError); # it must fall back to sequential like any other uncoercible value, not # crash the run. - results, _ = self._run(tmp_path, list(range(4)), bad) + results, _ = self._run(tmp_path, monkeypatch, list(range(4)), bad) assert results == [{"seen": i} for i in range(4)] - def test_string_max_concurrency_is_honored(self, tmp_path): - results, _ = self._run(tmp_path, list(range(4)), "2") + def test_string_max_concurrency_is_honored(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(4)), "2") assert results == [{"seen": i} for i in range(4)] - def test_context_item_isolation_across_threads(self, tmp_path): + def test_context_item_isolation_across_threads(self, tmp_path, monkeypatch): items = [{"id": f"x{i}"} for i in range(6)] - results, _ = self._run(tmp_path, items, 6) + results, _ = self._run(tmp_path, monkeypatch, items, 6) assert [r["seen"]["id"] for r in results] == [f"x{i}" for i in range(6)] @pytest.mark.parametrize("max_concurrency", [1, 2]) - def test_marks_item_context_as_inside_fan_out(self, tmp_path, max_concurrency): + def test_marks_item_context_as_inside_fan_out( + self, tmp_path, monkeypatch, max_concurrency + ): + from specify_cli.workflows import STEP_REGISTRY from specify_cli.workflows.base import StepBase, StepResult, StepStatus class _ContextProbeStep(StepBase): @@ -4508,28 +4511,27 @@ def execute(self, config, context): output={"inside_fan_out": context.inside_fan_out}, ) - engine, context, state, _registry, _template = self._build(tmp_path) - results = engine._run_fan_out( + monkeypatch.setitem(STEP_REGISTRY, "context-probe", _ContextProbeStep()) + results, _ = self._run( + tmp_path, + monkeypatch, ["a", "b"], - {"id": "probe", "type": "context-probe"}, - "fan", - context, - state, - {"context-probe": _ContextProbeStep()}, max_concurrency, + template={"id": "probe", "type": "context-probe"}, ) assert results == [ {"inside_fan_out": True}, {"inside_fan_out": True}, ] - assert context.inside_fan_out is False - def test_empty_items(self, tmp_path): - results, _ = self._run(tmp_path, [], 4) + def test_empty_items(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, [], 4) assert results == [] - def test_concurrent_halt_status_not_clobbered_by_later_item(self, tmp_path): + def test_concurrent_halt_status_not_clobbered_by_later_item( + self, tmp_path, monkeypatch + ): # Item 1 PAUSES (first halting item in order); item 3 FAILS while in # flight. The final run status must be the halting item's (PAUSED), never # a later item's (FAILED) that raced after it — matching sequential. @@ -4542,22 +4544,24 @@ def on_item(item): return StepStatus.FAILED return None - results, state = self._run(tmp_path, list(range(4)), 4, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(4)), 4, on_item) assert results == [{"seen": 0}, {"seen": 1}] assert state.status == RunStatus.PAUSED - def test_halt_on_failure_sequential_returns_prefix(self, tmp_path): + def test_halt_on_failure_sequential_returns_prefix(self, tmp_path, monkeypatch): from specify_cli.workflows.base import RunStatus, StepStatus def on_item(item): return StepStatus.FAILED if item == 2 else None - results, state = self._run(tmp_path, list(range(5)), 1, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(5)), 1, on_item) assert len(results) == 3 # items 0,1,2 ran; 3,4 never dispatched assert results[2] == {"seen": 2} assert state.status == RunStatus.FAILED - def test_halt_on_failure_concurrent_includes_halting_item(self, tmp_path): + def test_halt_on_failure_concurrent_includes_halting_item( + self, tmp_path, monkeypatch + ): # The concurrent prefix must match the sequential one: items up to and # INCLUDING the failing item (2), never a short prefix that drops it just # because a later in-flight item flipped the shared run status first. @@ -4566,11 +4570,11 @@ def test_halt_on_failure_concurrent_includes_halting_item(self, tmp_path): def on_item(item): return StepStatus.FAILED if item == 2 else None - results, state = self._run(tmp_path, list(range(6)), 4, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(6)), 4, on_item) assert results == [{"seen": 0}, {"seen": 1}, {"seen": 2}] assert state.status == RunStatus.FAILED - def test_concurrent_restores_halting_item_error(self, tmp_path): + def test_concurrent_restores_halting_item_error(self, tmp_path, monkeypatch): # After a concurrent fan-out halts, the run-level error must be the first # halting item's own error (parity with the sequential path), even when a # later concurrent item failed with a different error AND the halting @@ -4603,15 +4607,15 @@ def execute(self, config, context): status=StepStatus.COMPLETED, output={"seen": item} ) - engine, context, state, _registry, _template = self._build(tmp_path) - engine._run_fan_out( + from specify_cli.workflows import STEP_REGISTRY + + monkeypatch.setitem(STEP_REGISTRY, "err-probe", _ErrorProbe()) + _results, state = self._run( + tmp_path, + monkeypatch, ["ok0", "halt", "ok2", "leak"], - {"id": "impl", "type": "err-probe"}, - "fan", - context, - state, - {"err-probe": _ErrorProbe()}, 4, + template={"id": "impl", "type": "err-probe"}, ) assert state.status == RunStatus.FAILED @@ -4620,7 +4624,9 @@ def execute(self, config, context): # error verbatim, even when falsy. assert state.error == "" - def test_continue_on_error_item_does_not_halt_concurrent(self, tmp_path): + def test_continue_on_error_item_does_not_halt_concurrent( + self, tmp_path, monkeypatch + ): # A failing item whose template sets continue_on_error must NOT truncate # the fan-out: every item still runs and is returned in order. from specify_cli.workflows.base import StepStatus @@ -4628,41 +4634,38 @@ def test_continue_on_error_item_does_not_halt_concurrent(self, tmp_path): def on_item(item): return StepStatus.FAILED if item == 2 else None - engine, context, state, registry, template = self._build(tmp_path, on_item) - template["continue_on_error"] = True - results = engine._run_fan_out( - list(range(5)), template, "fan", context, state, registry, 4 + results, _ = self._run( + tmp_path, + monkeypatch, + list(range(5)), + 4, + on_item, + template={"id": "impl", "type": "probe", "continue_on_error": True}, ) assert results == [{"seen": i} for i in range(5)] - def test_unknown_template_type_halts_concurrent_like_sequential(self, tmp_path): + def test_unknown_template_type_halts_concurrent_like_sequential( + self, tmp_path, monkeypatch + ): # A template whose type isn't registered fails fast and records no result; # the concurrent path must still attribute the halt to the first item and # return the same prefix as sequential — never run on as if completed. - from specify_cli.workflows.base import RunStatus, StepContext - from specify_cli.workflows.engine import RunState, WorkflowEngine - - def fresh(): - state = RunState(run_id="r", workflow_id="w", project_root=tmp_path) - state.status = RunStatus.RUNNING - return WorkflowEngine(project_root=tmp_path), StepContext(), state - template = {"id": "impl", "type": "does-not-exist"} - e1, c1, s1 = fresh() - seq = e1._run_fan_out(list(range(5)), template, "fan", c1, s1, {}, 1) - e2, c2, s2 = fresh() - con = e2._run_fan_out(list(range(5)), template, "fan", c2, s2, {}, 4) + seq, s1 = self._run(tmp_path, monkeypatch, list(range(5)), 1, template=template) + con, s2 = self._run(tmp_path, monkeypatch, list(range(5)), 4, template=template) assert seq == con == [{}] # halted at the first item; rest never returned + from specify_cli.workflows.base import RunStatus + assert s1.status == s2.status == RunStatus.FAILED - def test_first_exception_cancels_and_reraises(self, tmp_path): + def test_first_exception_cancels_and_reraises(self, tmp_path, monkeypatch): def on_item(item): if item == 0: raise ValueError("boom") return None with pytest.raises(ValueError, match="boom"): - self._run(tmp_path, list(range(4)), 2, on_item) + self._run(tmp_path, monkeypatch, list(range(4)), 2, on_item) class TestFanInWaitForValidation: @@ -7739,6 +7742,34 @@ def test_unknown_step_type_sets_run_error(self, project_dir): assert state.status == RunStatus.FAILED assert state.error == "Unknown step type: 'definitely-not-a-real-step'" + def test_unknown_step_type_ignores_continue_on_error(self, project_dir): + """A missing implementation is terminal even with recovery enabled.""" + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + from specify_cli.workflows.base import RunStatus + + definition = WorkflowDefinition.from_string(""" +schema_version: "1.0" +workflow: + id: "unknown-type-recovery" + name: "Unknown Type Recovery" + version: "1.0.0" +steps: + - id: mystery + type: definitely-not-a-real-step + continue_on_error: true + - id: after + type: shell + run: "echo should-not-run" +""") + state = WorkflowEngine(project_dir).execute(definition) + + assert state.status == RunStatus.FAILED + assert state.error == "Unknown step type: 'definitely-not-a-real-step'" + assert "after" not in state.step_results + events = [entry["event"] for entry in state.log_entries] + assert "step_failed" in events + assert "step_continue_on_error" not in events + # ===== State Persistence Tests ===== diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index ae1bc88859..6639caec8d 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -290,6 +290,37 @@ def test_output_failure_retries_only_finalization(tmp_path, monkeypatch, probe): assert probe["work"] == 1 +@pytest.mark.parametrize("status", [RunStatus.PAUSED, RunStatus.FAILED]) +def test_resume_accepts_paused_and_failed_tree_backed_runs(tmp_path, probe, status): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "wait", "type": "probe", "status": status.value}], + ) + ) + assert state.status == status + + resumed = WorkflowEngine(tmp_path).resume(state.run_id) + + assert resumed.status == status + + +def test_resume_rejects_running_tree_backed_run_without_writes(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["status"] = RunStatus.RUNNING.value + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Cannot resume run .* 'running'"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert path.read_bytes() == before + + def test_legacy_resume_adapts_once(tmp_path, probe): root = definition( "parent", @@ -334,7 +365,7 @@ def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): @pytest.mark.parametrize("failure_after_replace", [False, True]) -def test_checkpoint_failure_never_overwrites_committed_progress( +def test_checkpoint_failure_leaves_running_run_not_resumable( tmp_path, monkeypatch, probe, failure_after_replace ): from specify_cli.workflows._execution import CheckpointError @@ -363,9 +394,11 @@ def write(path, data): node = disk["execution"]["sequence"]["nodes"][0] assert node["phase"] == ("done" if failure_after_replace else "ready") assert ("work" in disk["step_results"]) is failure_after_replace - state = WorkflowEngine(tmp_path).resume("fault") - assert state.status == RunStatus.COMPLETED - assert probe["work"] == (1 if failure_after_replace else 2) + with pytest.raises( + ValueError, match="Cannot resume run 'fault' with status 'running'" + ): + WorkflowEngine(tmp_path).resume("fault") + assert probe["work"] == 1 def test_completion_log_failure_does_not_replay_committed_step( @@ -440,6 +473,90 @@ def test_resolution_failures_are_call_failures(tmp_path, probe, mode): assert not probe +@pytest.mark.parametrize( + "child_continue, call_continue", [(False, False), (True, False), (False, True)] +) +def test_unknown_child_step_always_fails_despite_continue_on_error( + tmp_path, monkeypatch, probe, child_continue, call_continue +): + class TemporarilyInstalled(StepBase): + type_key = "temporarily-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "temporarily-installed", TemporarilyInstalled()) + install( + tmp_path, + definition( + "child", + [ + {"id": "wait", "type": "probe", "await": True}, + { + "id": "missing", + "type": "temporarily-installed", + "continue_on_error": child_continue, + } + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call( + input={"approve": "{{ inputs.approve }}"}, + continue_on_error=call_continue, + ), + {"id": "after", "type": "probe"}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + monkeypatch.delitem(STEP_REGISTRY, "temporarily-installed") + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.FAILED + assert state.error == "Unknown step type: 'temporarily-installed'" + assert probe == {"wait": 2} + events = [entry["event"] for entry in state.log_entries] + assert "step_failed" in events + assert "step_continue_on_error" not in events + + +def test_unknown_fan_out_template_step_always_fails_despite_continue_on_error( + tmp_path, probe +): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": { + "id": "missing", + "type": "not-installed", + "continue_on_error": True, + }, + }, + {"id": "after", "type": "probe"}, + ], + ) + ) + + assert state.status == RunStatus.FAILED + assert state.error == "Unknown step type: 'not-installed'" + assert not probe + events = [entry["event"] for entry in state.log_entries] + assert "step_failed" in events + assert "step_continue_on_error" not in events + + def test_diamond_and_depth_limit(tmp_path, probe): install(tmp_path, definition("leaf", [{"id": "work", "type": "probe"}])) for name in ("left", "right"): From 1d509d2ffcf709264249c9b1f0dbbb9626f2ead6 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:14:05 +0200 Subject: [PATCH 03/28] refactor(workflows): centralize execution tree access Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 125 ++++++++++-------- src/specify_cli/workflows/engine.py | 19 +-- tests/workflows/test_composition_execution.py | 23 ++++ 3 files changed, 103 insertions(+), 64 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index cc16871c5f..0cfd8d2948 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -32,13 +32,60 @@ def sequence(steps: list[dict[str, Any]]) -> dict[str, Any]: } +def new_execution( + steps: list[dict[str, Any]], offset: int, initial: dict[str, Any] +) -> dict[str, Any]: + """Create the persisted root execution tree.""" + return { + "version": 1, + "sequence": sequence(steps), + "offset": max(0, offset), + "initial": deepcopy(initial), + } + + +def steps_of(seq: dict[str, Any]) -> Any: + """Read the persisted YAML source for one execution sequence.""" + return yaml.safe_load(seq["source"]) + + +def child_path(config, path, item): + """Return a child's reporting path from its parent occurrence.""" + return ( + [*path, str(item)] + if config.get("type") in {"fan-out", "while", "do-while"} + else path + ) + + +def walk_execution(seq, path=(), *, children_first=False, skip_done=False): + """Yield execution occurrences with one shared path rule.""" + for index, (config, node) in enumerate(zip(steps_of(seq), seq["nodes"])): + if skip_done and node["phase"] == "done": + continue + here = [*path, config.get("id", f"step-{index}")] + children = node.get("children", []) + if children_first: + for item, child in enumerate(children): + yield from walk_execution( + child, + child_path(config, here, item), + children_first=True, + skip_done=skip_done, + ) + yield config, node, here + if not children_first: + for item, child in enumerate(children): + yield from walk_execution(child, child_path(config, here, item)) + + def validate_execution(tree: Any) -> None: """Validate stored structure without importing a project's custom steps.""" def check_sequence(seq): if not isinstance(seq, dict) or not isinstance(seq.get("source"), str): raise ValueError("Invalid execution sequence") - steps = yaml.safe_load(seq["source"]) + steps = steps_of(seq) nodes = seq.get("nodes") if ( not isinstance(steps, list) @@ -95,7 +142,7 @@ def check_sequence(seq): or not isinstance(binding.get("inputs"), dict) or len(children) != 1 or not isinstance(definition.get("steps"), list) - or definition["steps"] != yaml.safe_load(children[0]["source"]) + or definition["steps"] != steps_of(children[0]) ): raise ValueError("Invalid bound workflow definition or inputs") if node["phase"] == "outputs" and binding is None: @@ -128,57 +175,27 @@ def check_sequence(seq): def active_step(tree): """First unfinished leaf in execution order, including nested workflow scopes.""" - - def walk(seq, path): - for index, (config, node) in enumerate( - zip(yaml.safe_load(seq["source"]), seq["nodes"]) - ): - if node["phase"] == "done": - continue - here = [*path, config.get("id", f"step-{index}")] - for item, child in enumerate(node.get("children", [])): - child_path = ( - [*here, str(item)] - if config.get("type") in {"fan-out", "while", "do-while"} - else here - ) - found = walk(child, child_path) - if found: - return found - return here, node - return None - - return walk(tree["sequence"], []) + for _, node, path in walk_execution( + tree["sequence"], children_first=True, skip_done=True + ): + return path, node + return None def scope_summaries(tree): """Report workflow boundaries without exposing private inputs or results.""" summaries = [] - - def walk(seq, path): - for index, (config, node) in enumerate( - zip(yaml.safe_load(seq["source"]), seq["nodes"]) - ): - here = [*path, config.get("id", f"step-{index}")] - binding = node.get("binding") - if binding: - output = node.get("result", {}).get("output", {}) - summaries.append( - { - "scope_path": here, - "workflow_id": binding["workflow"], - "status": output.get("status", "running"), - } - ) - for item, child in enumerate(node.get("children", [])): - child_path = ( - [*here, str(item)] - if config.get("type") in {"fan-out", "while", "do-while"} - else here - ) - walk(child, child_path) - - walk(tree["sequence"], []) + for _, node, path in walk_execution(tree["sequence"]): + binding = node.get("binding") + if binding: + output = node.get("result", {}).get("output", {}) + summaries.append( + { + "scope_path": path, + "workflow_id": binding["workflow"], + "status": output.get("status", "running"), + } + ) return summaries @@ -217,7 +234,7 @@ def run( root=False, loop_alias=None, ): - steps = yaml.safe_load(seq["source"]) + steps = steps_of(seq) for index, (config, node) in enumerate(zip(steps, seq["nodes"])): config = {"id": f"step-{index}", **config} name = config["id"] @@ -246,7 +263,7 @@ def restore(self, node, context): """Rebuild aliases for completed expansions without re-executing them.""" for child in node.get("children", []) if "binding" not in node else []: for index, (config, nested) in enumerate( - zip(yaml.safe_load(child["source"]), child["nodes"]) + zip(steps_of(child), child["nodes"]) ): if "result" in nested: context.steps[config.get("id", f"step-{index}")] = nested["result"] @@ -363,7 +380,7 @@ def step(self, config, node, context, ancestry, path, public_name): while len(node["children"]) < limit and evaluate_condition( config.get("condition", False), context ): - child = sequence(yaml.safe_load(node["children"][0]["source"])) + child = sequence(steps_of(node["children"][0])) self.commit(node, {"children": [*node["children"], child]}) outcome, error = self.run( child, @@ -444,7 +461,7 @@ def finish(self, config, node, result, context, ancestry, path, public_name): return outcome def workflow(self, config, node, context, ancestry, path, public_name): - from .engine import WorkflowDefinition + from .engine import WorkflowDefinition, workflow_dir_for name = config.get("id", "step-0") self.log("step_started", name, path, ancestry[-1], type="workflow") @@ -464,9 +481,7 @@ def workflow(self, config, node, context, ancestry, path, public_name): "workflow": target, "definition": yaml.safe_dump(definition.data, sort_keys=False), "inputs": bind_inputs(self.engine, definition, config, context), - "workflow_dir": str(definition.source_path.parent) - if definition.source_path - else None, + "workflow_dir": workflow_dir_for(definition), } self.commit( node, diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 7fac4f7213..1764105708 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -117,6 +117,13 @@ def from_string(cls, content: str) -> WorkflowDefinition: return cls(data) +def workflow_dir_for(definition: WorkflowDefinition) -> str | None: + """Return the resolved parent directory of a workflow definition's source.""" + if definition.source_path is None: + return None + return str(definition.source_path.resolve().parent) + + # -- Workflow Validation -------------------------------------------------- # ID format: lowercase alphanumeric with hyphens @@ -1067,11 +1074,7 @@ def execute( # Resolve inputs resolved_inputs = self._resolve_inputs(definition, inputs or {}) state.inputs = resolved_inputs - workflow_dir = ( - str(definition.source_path.resolve().parent) - if definition.source_path is not None - else None - ) + workflow_dir = workflow_dir_for(definition) state.workflow_dir = workflow_dir state.status = RunStatus.RUNNING state.save() @@ -1229,13 +1232,11 @@ def _execute_steps( rebind: bool = False, ) -> None: """Execute or resume the persisted tree (legacy indices adapt once).""" - from ._execution import Execution, active_step, sequence + from ._execution import Execution, active_step, new_execution from copy import deepcopy if state.execution is None: - state.execution = {"version": 1, "sequence": sequence(steps)} - state.execution["offset"] = max(0, step_offset) - state.execution["initial"] = deepcopy(context.steps) + state.execution = new_execution(steps, step_offset, context.steps) context.steps = deepcopy(state.execution.get("initial", {})) state.save() tree = state.execution["sequence"] diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 6639caec8d..05e498e530 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -103,6 +103,29 @@ def test_declared_output_and_scope_isolation(tmp_path, probe): assert len(list((tmp_path / ".specify/workflows/runs").iterdir())) == 1 +def test_called_workflow_dir_resolves_definition_symlink(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + resolved_dir = tmp_path / "resolved-child" + resolved_dir.mkdir() + symlink_dir = tmp_path / "linked-child" + symlink_dir.symlink_to(resolved_dir, target_is_directory=True) + child = definition( + "child", + [{"id": "path", "type": "probe", "value": "{{ context.workflow_dir }}"}], + outputs={"workflow-dir": {"value": "{{ steps.path.output.value }}"}}, + ) + child.source_path = symlink_dir / "workflow.yml" + monkeypatch.setattr(execution, "resolve_target", lambda *_: child) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED, state.error + assert state.step_results["call"]["output"]["workflow-dir"] == str( + resolved_dir + ) + + def test_concurrent_nested_calls_keep_downstream_aliases_local( tmp_path, monkeypatch, probe ): From e487b534c5ca9a26181fd55964972471b13e216f Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:23:31 +0200 Subject: [PATCH 04/28] fix(workflows): narrow composition call failures Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 149 ++++---- src/specify_cli/workflows/composition.py | 116 ++++--- src/specify_cli/workflows/engine.py | 12 +- tests/workflows/test_composition_execution.py | 318 +++++++++++++++++- 4 files changed, 481 insertions(+), 114 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 0cfd8d2948..773af51b8a 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -15,7 +15,13 @@ import yaml from .base import StepContext, StepResult, StepStatus -from .composition import bind_inputs, evaluate_outputs, resolve_target, validate_call +from .composition import ( + CallError, + bind_inputs, + evaluate_outputs, + resolve_target, + validate_runtime_call, +) from .expressions import evaluate_condition, evaluate_expression HALTING = {"paused", "failed", "aborted"} @@ -199,6 +205,21 @@ def scope_summaries(tree): return summaries +def child_context_for_call(parent, binding, definition): + """Create the private context for one bound workflow call.""" + return StepContext( + inputs=binding["inputs"], + project_root=parent.project_root, + run_id=parent.run_id, + is_resume=parent.is_resume, + inside_fan_out=parent.inside_fan_out, + workflow_dir=binding["workflow_dir"], + default_integration=definition.default_integration, + default_model=definition.default_model, + default_options=definition.default_options, + ) + + class Execution: def __init__(self, engine, state, registry, *, rebind=False): self.engine, self.state, self.registry = engine, state, registry @@ -407,18 +428,27 @@ def step(self, config, node, context, ancestry, path, public_name): @staticmethod def record(config, result, context): output = result.output + is_workflow = config.get("type") == "workflow" data = { "type": config.get("type", "command"), - "integration": output.get("integration") + "integration": None + if is_workflow + else output.get("integration") or config.get("integration") or context.default_integration, - "model": output.get("model") + "model": None + if is_workflow + else output.get("model") or config.get("model") or context.default_model, - "options": output.get("options") or config.get("options", {}), - "input": {} - if config.get("type") == "workflow" - else output.get("input") or config.get("input", {}), + "options": {} + if is_workflow + else output.get("options") or config.get("options", {}), + "input": ( + {} + if is_workflow + else output.get("input") or config.get("input", {}) + ), "output": output, "status": result.status.value, "error": result.error, @@ -470,11 +500,9 @@ def workflow(self, config, node, context, ancestry, path, public_name): self.engine.on_step_start(name, "workflow") binding = node.get("binding") target = binding["workflow"] if binding else config.get("workflow") - try: - if binding is None: - errors = validate_call(config) - if errors: - raise ValueError("; ".join(errors)) + if binding is None: + try: + validate_runtime_call(config) target = evaluate_expression(target, context) definition = resolve_target(self.state.project_root, target, ancestry) binding = { @@ -491,56 +519,55 @@ def workflow(self, config, node, context, ancestry, path, public_name): "phase": "children", }, ) - else: - definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) - if self.rebind: - binding = { - **binding, - "inputs": bind_inputs(self.engine, definition, config, context), - } - self.commit(node, {"binding": binding}) - child_context = StepContext( - inputs=binding["inputs"], - project_root=context.project_root, - run_id=context.run_id, - is_resume=context.is_resume, - workflow_dir=binding["workflow_dir"], - default_integration=definition.default_integration, - default_model=definition.default_model, - default_options=definition.default_options, - ) - outcome, error = self.run( - node["children"][0], - child_context, - (*ancestry, target), - path=(*path, "workflow"), - public=False, - ) - output = {"workflow": target, "status": outcome} - if outcome == "completed": - self.commit(node, {"phase": "outputs"}) + except CallError as exc: + target = target if isinstance(target, str) else repr(target) + result = StepResult( + StepStatus.FAILED, + output={"workflow": target, "status": "failed", "error": str(exc)}, + error=str(exc), + ) + return self.finish( + config, node, result, context, ancestry, path, public_name + ) + else: + definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) + if self.rebind: + inputs = bind_inputs( + self.engine, definition, config, context, binding["inputs"] + ) + binding = {**binding, "inputs": inputs} + self.commit(node, {"binding": binding}) + child_context = child_context_for_call(context, binding, definition) + outcome, error = self.run( + node["children"][0], + child_context, + (*ancestry, target), + path=(*path, "workflow"), + public=False, + ) + output = {"workflow": target, "status": outcome} + if outcome == "completed": + self.commit(node, {"phase": "outputs"}) + try: output.update(evaluate_outputs(definition, child_context)) - elif outcome == "aborted": - output["aborted"] = True - if error is not None: - output["error"] = error - status = ( - StepStatus.COMPLETED - if outcome == "completed" - else StepStatus.PAUSED - if outcome == "paused" - else StepStatus.FAILED - ) - result = StepResult(status, output=output, error=error) - except CheckpointError: - raise - except Exception as exc: - target = target if isinstance(target, str) else repr(target) - result = StepResult( - StepStatus.FAILED, - output={"workflow": target, "status": "failed", "error": str(exc)}, - error=str(exc), - ) + except CallError as exc: + output.update(status="failed", error=str(exc)) + result = StepResult(StepStatus.FAILED, output=output, error=str(exc)) + return self.finish( + config, node, result, context, ancestry, path, public_name + ) + elif outcome == "aborted": + output["aborted"] = True + if error is not None: + output["error"] = error + status = ( + StepStatus.COMPLETED + if outcome == "completed" + else StepStatus.PAUSED + if outcome == "paused" + else StepStatus.FAILED + ) + result = StepResult(status, output=output, error=error) # An unavailable child implementation is terminal at every workflow # boundary, matching direct execution of the same step. if isinstance(result.error, str) and result.error.startswith( diff --git a/src/specify_cli/workflows/composition.py b/src/specify_cli/workflows/composition.py index 02c1156177..51db44e0c4 100644 --- a/src/specify_cli/workflows/composition.py +++ b/src/specify_cli/workflows/composition.py @@ -24,6 +24,10 @@ MAX_COMPOSITION_DEPTH = 16 +class CallError(ValueError): + """A treatable violation of the workflow-call boundary.""" + + def require_json(value: Any) -> None: """Reject lossy/non-JSON values, including cycles, before checkpointing.""" try: @@ -60,6 +64,13 @@ def validate_call(config: dict[str, Any]) -> list[str]: return errors +def validate_runtime_call(config: dict[str, Any]) -> None: + """Raise a call-boundary error for an invalid authored call shape.""" + errors = validate_call(config) + if errors: + raise CallError("; ".join(errors)) + + def validate_outputs(outputs: Any) -> list[str]: from .engine import _ID_PATTERN @@ -87,61 +98,84 @@ def resolve_target(project_root: Path, target: Any, ancestry: tuple[str, ...]): from .overlay import WorkflowResolver from .overlay.schema import _RESERVED_WORKFLOW_IDS - if not isinstance(target, str) or not _ID_PATTERN.fullmatch(target): - raise ValueError("Workflow target must be an exact safe workflow ID") - if target in _RESERVED_WORKFLOW_IDS: - raise ValueError(f"Workflow {target!r} is reserved") - if target in ancestry: - raise ValueError( - f"Workflow composition cycle: {' -> '.join((*ancestry, target))}" - ) - if len(ancestry) > MAX_COMPOSITION_DEPTH: - raise ValueError( - f"Workflow composition exceeds maximum depth {MAX_COMPOSITION_DEPTH}" - ) - metadata = WorkflowRegistry(project_root).get(target) - if not isinstance(metadata, dict): - raise ValueError(f"Workflow {target!r} is not installed") - if not metadata.get("enabled", True): - raise ValueError(f"Workflow {target!r} is disabled") - definition = WorkflowResolver(project_root).resolve(target) - if definition.id != target: - raise ValueError( - f"Workflow {target!r} resolves to mismatched ID {definition.id!r}" - ) - errors = validate_workflow(definition) - if errors: - raise ValueError(f"Invalid workflow {target!r}: {'; '.join(errors)}") - return definition - - -def bind_inputs(engine, definition, config: dict[str, Any], context: StepContext): + try: + if not isinstance(target, str) or not _ID_PATTERN.fullmatch(target): + raise CallError("Workflow target must be an exact safe workflow ID") + if target in _RESERVED_WORKFLOW_IDS: + raise CallError(f"Workflow {target!r} is reserved") + if target in ancestry: + raise CallError( + f"Workflow composition cycle: {' -> '.join((*ancestry, target))}" + ) + if len(ancestry) > MAX_COMPOSITION_DEPTH: + raise CallError( + f"Workflow composition exceeds maximum depth {MAX_COMPOSITION_DEPTH}" + ) + metadata = WorkflowRegistry(project_root).get(target) + if not isinstance(metadata, dict): + raise CallError(f"Workflow {target!r} is not installed") + if not metadata.get("enabled", True): + raise CallError(f"Workflow {target!r} is disabled") + definition = WorkflowResolver(project_root).resolve(target) + if definition.id != target: + raise CallError( + f"Workflow {target!r} resolves to mismatched ID {definition.id!r}" + ) + errors = validate_workflow(definition) + if errors: + raise CallError(f"Invalid workflow {target!r}: {'; '.join(errors)}") + return definition + except CallError: + raise + except Exception as exc: + raise CallError(str(exc)) from exc + + +def bind_inputs( + engine, + definition, + config: dict[str, Any], + context: StepContext, + previous: dict[str, Any] | None = None, +): mapping = config.get("input") if mapping is None: mapping = {} if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): - raise ValueError("'input' must be a mapping with string keys") + raise CallError("'input' must be a mapping with string keys") provided = { key: evaluate_expression(value, context) for key, value in mapping.items() } - require_json(provided) - unknown = provided.keys() - definition.inputs.keys() - if unknown: - raise ValueError( - f"Undeclared inputs for workflow {definition.id!r}: {sorted(unknown)}" + try: + require_json(provided) + unknown = provided.keys() - definition.inputs.keys() + if unknown: + raise CallError( + f"Undeclared inputs for workflow {definition.id!r}: {sorted(unknown)}" + ) + resolved = ( + engine._resolve_inputs(definition, provided) + if previous is None + else engine._merge_inputs(definition, previous, provided) ) - resolved = engine._resolve_inputs(definition, provided) - require_json(resolved) - return resolved + require_json(resolved) + return resolved + except CallError: + raise + except (TypeError, ValueError, RecursionError) as exc: + raise CallError(str(exc)) from exc def evaluate_outputs(definition, context: StepContext) -> dict[str, Any]: errors = validate_outputs(definition.outputs) if errors: - raise ValueError("; ".join(errors)) + raise CallError("; ".join(errors)) output = { name: evaluate_expression(entry["value"], context) for name, entry in definition.outputs.items() } - require_json(output) - return output + try: + require_json(output) + return output + except (TypeError, ValueError, RecursionError) as exc: + raise CallError(str(exc)) from exc diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 1764105708..96f215a789 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -1168,8 +1168,7 @@ def resume( # Merge any newly-supplied inputs over the persisted ones and # re-validate through the same typing path as the initial run. if inputs: - merged = {**state.inputs, **inputs} - state.inputs = self._resolve_inputs(definition, merged) + state.inputs = self._merge_inputs(definition, state.inputs, inputs) # Restore context context = StepContext( @@ -1313,6 +1312,15 @@ def _resolve_inputs( resolved[name] = self._coerce_input(name, value, coerce_input_def) return resolved + def _merge_inputs( + self, + definition: WorkflowDefinition, + previous: dict[str, Any], + updates: dict[str, Any], + ) -> dict[str, Any]: + """Resolve updates over an existing input snapshot.""" + return self._resolve_inputs(definition, {**previous, **updates}) + def _resolve_default(self, name: str, default: Any) -> Any: """Resolve special default sentinels against project state. diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 05e498e530..0d740fe5a7 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -293,18 +293,79 @@ def test_invalid_inputs_do_not_execute_child(tmp_path, probe, mapping): RunState.load(state.run_id, tmp_path) -def test_output_failure_retries_only_finalization(tmp_path, monkeypatch, probe): +def test_child_exception_propagates_despite_continue_on_error(tmp_path, monkeypatch): + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise RuntimeError("child exploded") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + install(tmp_path, definition("child", [{"id": "work", "type": "explode"}])) + + with pytest.raises(RuntimeError, match="child exploded"): + WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + + +@pytest.mark.parametrize("location", ["input", "output"]) +def test_call_expression_errors_propagate_despite_continue_on_error( + tmp_path, probe, location +): + child = definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string"}}, + outputs=( + {"value": {"value": "{{ inputs.value | from_json }}"}} + if location == "output" + else {} + ), + ) + install(tmp_path, child) + config = call( + input={ + "value": ( + "{{ inputs.value | from_json }}" if location == "input" else "not json" + ) + }, + continue_on_error=True, + ) + + with pytest.raises(ValueError, match="from_json: invalid JSON"): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [config], + inputs={"value": {"type": "string", "default": "not json"}}, + ) + ) + + +@pytest.mark.parametrize("handled", [False, True]) +def test_output_failure_retries_only_finalization( + tmp_path, monkeypatch, probe, handled +): import specify_cli.workflows._execution as execution + from specify_cli.workflows.composition import CallError install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) original = execution.evaluate_outputs monkeypatch.setattr( execution, "evaluate_outputs", - lambda *_: (_ for _ in ()).throw(ValueError("bad output")), + lambda *_: (_ for _ in ()).throw(CallError("bad output")), ) - state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) - assert state.status == RunStatus.FAILED + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=handled)]) + ) + assert state.status == ( + RunStatus.COMPLETED if handled else RunStatus.FAILED + ) + if handled: + assert probe["work"] == 1 + return state = WorkflowEngine(tmp_path).resume(state.run_id) assert state.status == RunStatus.FAILED monkeypatch.setattr(execution, "evaluate_outputs", original) @@ -313,6 +374,35 @@ def test_output_failure_retries_only_finalization(tmp_path, monkeypatch, probe): assert probe["work"] == 1 +def test_output_expression_failure_retries_only_finalization(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string"}}, + outputs={"value": {"value": "{{ inputs.value | from_json }}"}}, + ), + ) + root = definition( + "parent", + [call(input={"value": "{{ inputs.value }}"})], + inputs={"value": {"type": "string", "default": "not json"}}, + ) + + with pytest.raises(ValueError, match="from_json: invalid JSON"): + WorkflowEngine(tmp_path).execute(root, run_id="output-expression") + + state = WorkflowEngine(tmp_path).resume( + "output-expression", + {"value": '{"ok": true}'}, + ) + + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + assert state.step_results["call"]["output"]["value"] == {"ok": True} + + @pytest.mark.parametrize("status", [RunStatus.PAUSED, RunStatus.FAILED]) def test_resume_accepts_paused_and_failed_tree_backed_runs(tmp_path, probe, status): state = WorkflowEngine(tmp_path).execute( @@ -673,14 +763,222 @@ def test_rebind_failure_has_one_failed_caller_outcome(tmp_path, probe): ) state = WorkflowEngine(tmp_path).execute(root) assert state.status == RunStatus.PAUSED - state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "invalid"}) + before = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0] + + with pytest.raises(ValueError, match="expected a boolean"): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "invalid"}) + + failed = RunState.load(state.run_id, tmp_path) + node = failed.execution["sequence"]["nodes"][0] + assert failed.status == RunStatus.FAILED + assert node == before + assert WorkflowEngine(tmp_path).resume( + state.run_id, {"approve": "true"} + ).status == RunStatus.COMPLETED + + +@pytest.mark.parametrize("depth", [1, 2]) +def test_verdict_input_remains_forbidden_through_calls_in_fan_out( + tmp_path, probe, depth +): + child = definition( + "child", + [ + { + "id": "review", + "type": "gate", + "message": "Review", + "verdict_input": "approve", + } + ], + inputs={"approve": {"type": "string", "default": ""}}, + ) + install(tmp_path, child) + if depth == 2: + install( + tmp_path, + definition( + "middle", + [call("child", input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "string", "default": ""}}, + ), + ) + target = "middle" if depth == 2 else "child" + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": ["item"], + "step": { + "id": "call", + "type": "workflow", + "workflow": target, + "input": {"approve": ""}, + }, + } + ], + ) + ) + + assert state.status == RunStatus.FAILED + assert "not supported inside fan-out templates" in state.error + + +def test_workflow_result_does_not_inherit_parent_defaults(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + workflow={ + "id": "child", + "name": "child", + "integration": "child", + "model": "child-model", + "options": {"x": 2}, + }, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call()], + workflow={ + "id": "parent", + "name": "parent", + "integration": "parent", + "model": "parent-model", + "options": {"x": 1}, + }, + ) + ) + + call_result = state.step_results["call"] + assert call_result["integration"] is None + assert call_result["model"] is None + assert call_result["options"] == {} + assert call_result["input"] == {} + + +def test_child_context_does_not_inherit_item_or_fan_in(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [ + {"id": "item", "type": "probe", "value": "{{ item }}"}, + {"id": "fan-in", "type": "probe", "value": "{{ fan_in }}"}, + ], + outputs={ + "item": {"value": "{{ steps.item.output.value }}"}, + "fan-in": {"value": "{{ steps.fan-in.output.value }}"}, + }, + ), + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"] == { + "workflow": "child", + "status": "completed", + "item": None, + "fan-in": {}, + } + + +def test_rebind_keeps_unmapped_auto_input(tmp_path, probe): + marker = tmp_path / ".specify" / "integration.json" + marker.parent.mkdir() + marker.write_text('{"version": 1, "default_integration": "first"}') + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={ + "approve": {"type": "boolean", "default": False}, + "integration": {"type": "string", "default": "auto"}, + }, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + marker.write_text('{"version": 1, "default_integration": "second"}') + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED - node = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0] - assert node["phase"] == "done" - assert node["result"]["output"]["status"] == "failed" - from specify_cli.workflows._execution import active_step + binding = state.execution["sequence"]["nodes"][0]["binding"] + assert binding["inputs"] == {"approve": True, "integration": "first"} + + +@pytest.mark.parametrize("change", ["disable", "uninstall"]) +def test_bound_call_uses_snapshot_after_target_changes(tmp_path, probe, change): + child = definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + directory = install(tmp_path, child) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + if change == "disable": + from specify_cli.workflows.catalog import WorkflowRegistry + + WorkflowRegistry(tmp_path).add("child", {"version": "1.0.0", "enabled": False}) + else: + directory.rename(tmp_path / "removed-child") + + resumed = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert resumed.status == RunStatus.COMPLETED + - assert active_step(state.execution) is None +@pytest.mark.parametrize("change", ["disable", "uninstall"]) +def test_unbound_call_checks_target_when_resume_reaches_it(tmp_path, probe, change): + child = definition("child", [{"id": "work", "type": "probe"}]) + directory = install(tmp_path, child) + root = definition( + "parent", + [ + { + "id": "wait", + "type": "gate", + "message": "Wait", + "verdict_input": "approve", + }, + call(continue_on_error=True), + ], + inputs={"approve": {"type": "string", "default": ""}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + if change == "disable": + from specify_cli.workflows.catalog import WorkflowRegistry + + WorkflowRegistry(tmp_path).add("child", {"version": "1.0.0", "enabled": False}) + else: + directory.rename(tmp_path / "removed-child") + + resumed = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "approve"}) + + assert resumed.status == RunStatus.COMPLETED + assert resumed.step_results["call"]["status"] == "failed" + assert not probe @pytest.mark.parametrize( From cc8e9de513d68d43821e6ee5825b5a77db1af006 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:34:43 +0200 Subject: [PATCH 05/28] fix(workflows): replay execution projections Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 197 +++++++++++------ tests/workflows/test_composition_execution.py | 200 ++++++++++++++++++ 2 files changed, 336 insertions(+), 61 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 773af51b8a..dbca3979d1 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -220,12 +220,43 @@ def child_context_for_call(parent, binding, definition): ) +def qualified_id(name, loop_alias=None): + """Return the public occurrence ID for a direct loop body step.""" + if loop_alias is None: + return name + return f"{loop_alias[0]}:{name}:{loop_alias[1]}" + + class Execution: def __init__(self, engine, state, registry, *, rebind=False): self.engine, self.state, self.registry = engine, state, registry self.rebind = rebind - def commit(self, node=None, changes=None, *, context=None, name=None, public=None): + def project(self, context, name, result, *, public, qualified): + """Apply an occurrence result to its local and public views.""" + context.steps[name] = result + if public: + self.state.step_results[name] = result + if qualified != name: + context.steps[qualified] = result + self.state.step_results[qualified] = result + + def project_alias(self, context, name, result): + """Project a fan-out item result without exposing item internals.""" + context.steps[name] = result + self.state.step_results[name] = result + + def commit( + self, + node=None, + changes=None, + *, + context=None, + name=None, + public=False, + qualified=None, + aliases=(), + ): """Mutate an occurrence and its compatibility views in one checkpoint.""" with self.state._lock: if self.state._checkpoint_failed: @@ -233,9 +264,15 @@ def commit(self, node=None, changes=None, *, context=None, name=None, public=Non if node is not None: node.update(changes or {}) if context is not None and "result" in node: - context.steps[name] = node["result"] - if public is not None: - self.state.step_results[public] = node["result"] + self.project( + context, + name, + node["result"], + public=public, + qualified=qualified or name, + ) + for alias_context, alias, result in aliases: + self.project_alias(alias_context, alias, result) self.state.save() def log(self, event, name, path, workflow, **fields): @@ -260,7 +297,7 @@ def run( config = {"id": f"step-{index}", **config} name = config["id"] occurrence = (*path, index) - public_name = name if public else None + qualified = qualified_id(name, loop_alias) if root and node["phase"] != "done": with self.state._lock: self.state.current_step_index = index + self.state.execution.get( @@ -268,50 +305,75 @@ def run( ) self.state.current_step_id = name outcome = self.step( - config, node, context, ancestry, occurrence, public_name + config, node, context, ancestry, occurrence, public, qualified ) - if loop_alias is not None and "result" in node: - with self.state._lock: - key = f"{loop_alias[0]}:{name}:{loop_alias[1]}" - self.state.step_results[key] = node["result"] - context.steps[key] = node["result"] - self.state.save() if outcome in HALTING: return outcome, node.get("error") return "completed", None - def restore(self, node, context): - """Rebuild aliases for completed expansions without re-executing them.""" - for child in node.get("children", []) if "binding" not in node else []: - for index, (config, nested) in enumerate( - zip(steps_of(child), child["nodes"]) - ): - if "result" in nested: - context.steps[config.get("id", f"step-{index}")] = nested["result"] - if config.get("type") != "fan-out": - self.restore(nested, context) + def replay_fan_out(self, config, node, context, ancestry, path, public, qualified): + """Replay persisted fan-out items in order without starting workers.""" + output = node["result"]["output"] + initial = deepcopy(context.steps) + items = output.get("items", []) + template_name = output.get("step_template", {}).get("id", "item") + for index, child in enumerate(node.get("children", [])): + local = replace( + context, + steps=deepcopy(initial), + item=items[index], + inside_fan_out=True, + ) + outcome, error = self.run( + child, local, ancestry, path=(*path, "item", index), public=False + ) + record = child["nodes"][0].get("result") + if record is not None: + self.project_alias( + context, + f"{qualified}:{template_name}:{index}", + record, + ) + if outcome in HALTING: + return outcome, error + return "completed", None - def step(self, config, node, context, ancestry, path, public_name): + def step(self, config, node, context, ancestry, path, public, qualified): name = config.get("id", "step-0") kind = config.get("type", "command") if node["phase"] == "done": - context.steps[name] = node["result"] + self.project( + context, name, node["result"], public=public, qualified=qualified + ) + if kind == "workflow": + return node.get("outcome", "completed") if kind == "fan-out": - template = node["result"]["output"].get("step_template", {}) - for index, child in enumerate(node.get("children", [])): - result = child["nodes"][0].get("result") - if result is not None: - context.steps[ - f"{name}:{template.get('id', 'item')}:{index}" - ] = result + outcome, _ = self.replay_fan_out( + config, node, context, ancestry, path, public, qualified + ) else: - self.restore(node, context) - return node.get("outcome", "completed") + outcome = "completed" + for iteration, child in enumerate(node.get("children", [])): + outcome, _ = self.run( + child, + context, + ancestry, + path=(*path, iteration), + public=public, + loop_alias=(qualified, iteration) + if kind in {"while", "do-while"} and iteration + else None, + ) + if outcome in HALTING: + break + return outcome if outcome in HALTING else node.get("outcome", "completed") if node.get("outcome") == "aborted": return "aborted" if kind == "workflow": - return self.workflow(config, node, context, ancestry, path, public_name) + return self.workflow( + config, node, context, ancestry, path, public, qualified + ) if node["phase"] in {"ready", "blocked"}: with self.state._lock: @@ -334,12 +396,13 @@ def step(self, config, node, context, ancestry, path, public_name): context, ancestry, path, - public_name, + public, + qualified, ) result = impl.execute(config, context) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: return self.finish( - config, node, result, context, ancestry, path, public_name + config, node, result, context, ancestry, path, public, qualified ) children = [sequence(result.next_steps)] if result.next_steps else [] if kind == "fan-out": @@ -354,28 +417,36 @@ def step(self, config, node, context, ancestry, path, public_name): if kind == "fan-out": result.output = {**result.output, "results": []} return self.finish( - config, node, result, context, ancestry, path, public_name + config, node, result, context, ancestry, path, public, qualified ) self.commit( node, {"phase": "children", "result": data, "children": children}, context=context, name=name, - public=public_name, + public=public, + qualified=qualified, ) else: - context.steps[name] = node["result"] + self.project( + context, name, node["result"], public=public, qualified=qualified + ) if kind == "fan-out": outcome, error, outputs = self.fan_out( - config, node, context, ancestry, path, public_name + config, node, context, ancestry, path, public, qualified ) data = { **node["result"], "output": {**node["result"]["output"], "results": outputs}, } self.commit( - node, {"result": data}, context=context, name=name, public=public_name + node, + {"result": data}, + context=context, + name=name, + public=public, + qualified=qualified, ) else: outcome, error = "completed", None @@ -385,11 +456,10 @@ def step(self, config, node, context, ancestry, path, public_name): context, ancestry, path=(*path, iteration), - public=public_name is not None, - loop_alias=(name, iteration) + public=public, + loop_alias=(qualified, iteration) if kind in {"while", "do-while"} and iteration - and public_name is not None else None, ) if outcome in HALTING: @@ -408,10 +478,8 @@ def step(self, config, node, context, ancestry, path, public_name): context, ancestry, path=(*path, len(node["children"]) - 1), - public=public_name is not None, - loop_alias=(name, len(node["children"]) - 1) - if public_name is not None - else None, + public=public, + loop_alias=(qualified, len(node["children"]) - 1), ) if outcome in HALTING: break @@ -460,7 +528,7 @@ def record(config, result, context): ) return data - def finish(self, config, node, result, context, ancestry, path, public_name): + def finish(self, config, node, result, context, ancestry, path, public, qualified): name = config.get("id", "step-0") outcome = "completed" if result.status == StepStatus.PAUSED: @@ -479,7 +547,8 @@ def finish(self, config, node, result, context, ancestry, path, public_name): }, context=context, name=name, - public=public_name, + public=public, + qualified=qualified, ) self.log("step_completed", name, path, ancestry[-1], status=result.status.value) if result.status == StepStatus.FAILED: @@ -490,7 +559,7 @@ def finish(self, config, node, result, context, ancestry, path, public_name): self.log(event, name, path, ancestry[-1], error=result.error) return outcome - def workflow(self, config, node, context, ancestry, path, public_name): + def workflow(self, config, node, context, ancestry, path, public, qualified): from .engine import WorkflowDefinition, workflow_dir_for name = config.get("id", "step-0") @@ -527,7 +596,7 @@ def workflow(self, config, node, context, ancestry, path, public_name): error=str(exc), ) return self.finish( - config, node, result, context, ancestry, path, public_name + config, node, result, context, ancestry, path, public, qualified ) else: definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) @@ -554,7 +623,7 @@ def workflow(self, config, node, context, ancestry, path, public_name): output.update(status="failed", error=str(exc)) result = StepResult(StepStatus.FAILED, output=output, error=str(exc)) return self.finish( - config, node, result, context, ancestry, path, public_name + config, node, result, context, ancestry, path, public, qualified ) elif outcome == "aborted": output["aborted"] = True @@ -574,9 +643,11 @@ def workflow(self, config, node, context, ancestry, path, public_name): "Unknown step type: " ): config = {**config, "continue_on_error": False} - return self.finish(config, node, result, context, ancestry, path, public_name) + return self.finish( + config, node, result, context, ancestry, path, public, qualified + ) - def fan_out(self, config, node, context, ancestry, path, public_name): + def fan_out(self, config, node, context, ancestry, path, public, qualified): output = node["result"]["output"] items = output.get("items", []) try: @@ -599,12 +670,16 @@ def run_item(index): halted.set() record = child["nodes"][0].get("result", {}) template_name = output.get("step_template", {}).get("id", "item") - with self.state._lock: - key = f"{config['id']}:{template_name}:{index}" - context.steps[key] = record - if public_name is not None: - self.state.step_results[key] = record - self.state.save() + if public: + self.commit( + aliases=[ + ( + context, + f"{qualified}:{template_name}:{index}", + record, + ) + ] + ) return outcome, error, record.get("output", {}) results = [] diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 0d740fe5a7..20a4512e5c 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -567,6 +567,206 @@ def test_expansion_resume_preserves_completed_work(tmp_path, probe, kind): assert probe["prepare"] == (1 if kind == "if" else 2) +def test_replay_restores_fan_out_aliases_from_completed_if(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + } + ], + }, + {"id": "wait", "type": "probe", "await": True}, + {"id": "join", "type": "fan-in", "wait_for": ["fan:template:0"]}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["join"]["output"]["results"] == [{"value": 1}] + + +def test_replay_does_not_execute_completed_fan_out_or_emit_callbacks(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + root = definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + assert probe == {"template": 2, "wait": 1} + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"template": 2, "wait": 2} + assert callbacks == ["wait"] + + +def test_replay_does_not_evaluate_completed_loop_condition( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": False, + "steps": [{"id": "work", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + monkeypatch.setattr( + execution, + "evaluate_condition", + lambda *_: pytest.fail("replay must not evaluate loop conditions"), + ) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"work": 1, "wait": 2} + + +def test_replay_does_not_commit_completed_nodes(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "done", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + original = execution.Execution.commit + committed = [] + + def commit(self, node=None, changes=None, **kwargs): + if node is not None and node["phase"] == "done": + committed.append(node) + return original(self, node, changes, **kwargs) + + monkeypatch.setattr(execution.Execution, "commit", commit) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert committed == [] + + +def test_private_fan_out_context_has_no_qualified_aliases(tmp_path, monkeypatch, probe): + class Inspect(StepBase): + type_key = "inspect" + + def execute(self, config, context): + return StepResult( + StepStatus.COMPLETED, + output={"aliases": sorted(key for key in context.steps if ":" in key)}, + ) + + monkeypatch.setitem(STEP_REGISTRY, "inspect", Inspect()) + install( + tmp_path, + definition( + "child", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe"}, + }, + {"id": "inspect", "type": "inspect"}, + ], + outputs={"aliases": {"value": "{{ steps.inspect.output.aliases }}"}}, + ), + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"]["aliases"] == [] + + +def test_fan_out_in_later_loop_iteration_uses_qualified_aliases(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + } + ], + } + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan:template:0"]["output"]["value"] == 1 + assert state.step_results["fan:template:1"]["output"]["value"] == 2 + assert state.step_results["loop:fan:1:template:0"]["output"]["value"] == 1 + assert state.step_results["loop:fan:1:template:1"]["output"]["value"] == 2 + + @pytest.mark.parametrize("mode", ["unknown", "disabled", "mismatch", "cycle"]) def test_resolution_failures_are_call_failures(tmp_path, probe, mode): if mode != "unknown": From b84ec98b6794494ec0df7d4169448c1ae51906e9 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:40:45 +0200 Subject: [PATCH 06/28] fix(workflows): qualify execution events Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 87 ++++-- tests/workflows/test_composition_execution.py | 261 ++++++++++++++++++ 2 files changed, 332 insertions(+), 16 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index dbca3979d1..46a43d07e8 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -275,11 +275,25 @@ def commit( self.project_alias(alias_context, alias, result) self.state.save() - def log(self, event, name, path, workflow, **fields): - entry = {"event": event, "step_id": name, **fields} - if path: + def emit( + self, + event, + qualified, + path, + workflow, + *, + private=False, + callback_label=None, + **fields, + ): + """Emit one step event and its optional start callback.""" + entry = {"event": event, "step_id": qualified, **fields} + if private: entry.update(execution_path=list(path), workflow_id=workflow) self.state.append_log(entry) + if callback_label is not None and self.engine.on_step_start is not None: + with self.engine._callback_lock: + self.engine.on_step_start(qualified, callback_label) def run( self, @@ -325,7 +339,12 @@ def replay_fan_out(self, config, node, context, ancestry, path, public, qualifie inside_fan_out=True, ) outcome, error = self.run( - child, local, ancestry, path=(*path, "item", index), public=False + child, + local, + ancestry, + path=(*path, "item", index), + public=False, + loop_alias=(qualified, index), ) record = child["nodes"][0].get("result") if record is not None: @@ -378,10 +397,15 @@ def step(self, config, node, context, ancestry, path, public, qualified): if node["phase"] in {"ready", "blocked"}: with self.state._lock: self.state.current_step_id = name - self.log("step_started", name, path, ancestry[-1], type=kind) - if self.engine.on_step_start is not None: - with self.engine._callback_lock: - self.engine.on_step_start(name, config.get("command", "") or kind) + self.emit( + "step_started", + qualified, + path, + ancestry[-1], + private=len(ancestry) > 1, + type=kind, + callback_label=config.get("command", "") or kind, + ) impl = self.registry.get(kind) if impl is None: # A missing implementation is a runtime configuration error, not @@ -427,6 +451,14 @@ def step(self, config, node, context, ancestry, path, public, qualified): public=public, qualified=qualified, ) + self.emit( + "step_completed", + qualified, + path, + ancestry[-1], + private=len(ancestry) > 1, + status=result.status.value, + ) else: self.project( context, name, node["result"], public=public, qualified=qualified @@ -550,23 +582,41 @@ def finish(self, config, node, result, context, ancestry, path, public, qualifie public=public, qualified=qualified, ) - self.log("step_completed", name, path, ancestry[-1], status=result.status.value) + self.emit( + "step_completed", + qualified, + path, + ancestry[-1], + private=len(ancestry) > 1, + status=result.status.value, + ) if result.status == StepStatus.FAILED: event = { "aborted": "workflow_aborted", "completed": "step_continue_on_error", }.get(outcome, "step_failed") - self.log(event, name, path, ancestry[-1], error=result.error) + self.emit( + event, + qualified, + path, + ancestry[-1], + private=len(ancestry) > 1, + error=result.error, + ) return outcome def workflow(self, config, node, context, ancestry, path, public, qualified): from .engine import WorkflowDefinition, workflow_dir_for - name = config.get("id", "step-0") - self.log("step_started", name, path, ancestry[-1], type="workflow") - if self.engine.on_step_start is not None: - with self.engine._callback_lock: - self.engine.on_step_start(name, "workflow") + self.emit( + "step_started", + qualified, + path, + ancestry[-1], + private=len(ancestry) > 1, + type="workflow", + callback_label="workflow", + ) binding = node.get("binding") target = binding["workflow"] if binding else config.get("workflow") if binding is None: @@ -664,7 +714,12 @@ def run_item(index): ) child = node["children"][index] outcome, error = self.run( - child, local, ancestry, path=(*path, "item", index), public=False + child, + local, + ancestry, + path=(*path, "item", index), + public=False, + loop_alias=(qualified, index), ) if outcome in HALTING: halted.set() diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 20a4512e5c..10079af431 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -537,6 +537,50 @@ def log(self, entry): assert probe["work"] == 1 +def test_container_completion_log_failure_does_not_replay_expansion( + tmp_path, monkeypatch, probe +): + original = RunState.append_log + failed = False + expanded = 0 + + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + nonlocal expanded + expanded += 1 + return StepResult( + StepStatus.COMPLETED, + next_steps=[{"id": "work", "type": "probe"}], + ) + + def log(self, entry): + nonlocal failed + if entry == { + "event": "step_completed", + "step_id": "expand", + "status": "completed", + } and not failed: + failed = True + raise OSError("log failed") + original(self, entry) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + monkeypatch.setattr(RunState, "append_log", log) + with pytest.raises(OSError, match="log failed"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "expand", "type": "expand"}]), + run_id="container-log", + ) + + state = WorkflowEngine(tmp_path).resume("container-log") + + assert state.status == RunStatus.COMPLETED + assert expanded == 1 + assert probe["work"] == 1 + + @pytest.mark.parametrize("kind", ["if", "while", "do-while", "fan-out"]) def test_expansion_resume_preserves_completed_work(tmp_path, probe, kind): body = [ @@ -634,6 +678,11 @@ def test_replay_does_not_execute_completed_fan_out_or_emit_callbacks(tmp_path, p assert state.status == RunStatus.COMPLETED assert probe == {"template": 2, "wait": 2} assert callbacks == ["wait"] + assert [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] == [("step_started", "wait"), ("step_completed", "wait")] def test_replay_does_not_evaluate_completed_loop_condition( @@ -767,6 +816,218 @@ def test_fan_out_in_later_loop_iteration_uses_qualified_aliases(tmp_path, probe) assert state.step_results["loop:fan:1:template:1"]["output"]["value"] == 2 +def test_fan_out_events_and_callbacks_use_qualified_item_ids(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "template", "type": "probe"}, + } + ], + ) + ) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert callbacks == ["fan", "fan:template:0", "fan:template:1"] + assert events == [ + ("step_started", "fan"), + ("step_completed", "fan"), + ("step_started", "fan:template:0"), + ("step_completed", "fan:template:0"), + ("step_started", "fan:template:1"), + ("step_completed", "fan:template:1"), + ] + + +def test_loop_events_and_callbacks_use_qualified_iteration_ids(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + } + ], + ) + ) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert callbacks == ["loop", "body", "loop:body:1"] + assert events == [ + ("step_started", "loop"), + ("step_completed", "loop"), + ("step_started", "body"), + ("step_completed", "body"), + ("step_started", "loop:body:1"), + ("step_completed", "loop:body:1"), + ] + + +def test_private_scope_events_include_execution_path_and_workflow_id(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + state = engine.execute(definition("parent", [call()])) + + call_entries = [ + entry + for entry in state.log_entries + if entry.get("step_id") == "call" and entry["event"].startswith("step_") + ] + child_entries = [ + entry + for entry in state.log_entries + if entry.get("step_id") == "work" and entry["event"].startswith("step_") + ] + assert all("execution_path" not in entry for entry in call_entries) + assert all("workflow_id" not in entry for entry in call_entries) + assert all(entry["workflow_id"] == "child" for entry in child_entries) + assert all(entry["execution_path"] == [0, "workflow", 0] for entry in child_entries) + assert callbacks == ["call", "work"] + + +def test_completed_workflow_call_replay_emits_no_events(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + root = definition( + "parent", + [ + call(), + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert state.status == RunStatus.COMPLETED + assert callbacks == ["wait"] + assert events == [("step_started", "wait"), ("step_completed", "wait")] + + +def test_unfinished_workflow_call_emits_events_on_resume(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert state.status == RunStatus.COMPLETED + assert callbacks == ["call", "wait"] + assert events == [ + ("step_started", "call"), + ("step_started", "wait"), + ("step_completed", "wait"), + ("step_completed", "call"), + ] + + +@pytest.mark.parametrize( + "template, expected_event, status", + [ + ({"status": "failed"}, "step_failed", RunStatus.FAILED), + ( + {"status": "failed", "continue_on_error": True}, + "step_continue_on_error", + RunStatus.COMPLETED, + ), + ( + {"status": "failed", "output": {"aborted": True}}, + "workflow_aborted", + RunStatus.ABORTED, + ), + ], +) +def test_fan_out_failure_events_use_qualified_item_ids( + tmp_path, probe, template, expected_event, status +): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe", **template}, + } + ], + ) + ) + + failures = [ + entry + for entry in state.log_entries + if entry["event"] in { + "step_failed", + "step_continue_on_error", + "workflow_aborted", + } + ] + assert state.status == status + assert len(failures) == 1 + assert failures[0]["step_id"] == "fan:template:0" + assert failures[0]["event"] == expected_event + + @pytest.mark.parametrize("mode", ["unknown", "disabled", "mismatch", "cycle"]) def test_resolution_failures_are_call_failures(tmp_path, probe, mode): if mode != "unknown": From 8a431644b9bdbc3bda32fa63f1e0ce4027678772 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:51:11 +0200 Subject: [PATCH 07/28] fix(workflows): share execution snapshots Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 211 ++++++++++++++---- src/specify_cli/workflows/engine.py | 6 +- tests/workflows/test_composition_execution.py | 209 +++++++++++++++++ 3 files changed, 381 insertions(+), 45 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 46a43d07e8..db64d843f6 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -38,6 +38,11 @@ def sequence(steps: list[dict[str, Any]]) -> dict[str, Any]: } +def occurrences(steps: list[dict[str, Any]]) -> dict[str, Any]: + """Create occurrence storage that shares its parent's immutable source.""" + return {"nodes": [{"phase": "ready"} for _ in steps]} + + def new_execution( steps: list[dict[str, Any]], offset: int, initial: dict[str, Any] ) -> dict[str, Any]: @@ -55,6 +60,18 @@ def steps_of(seq: dict[str, Any]) -> Any: return yaml.safe_load(seq["source"]) +def child_steps(config, node, child): + """Read a child sequence from its position in the parent occurrence.""" + kind = config.get("type") + if kind == "workflow": + return yaml.safe_load(node["binding"]["definition"])["steps"] + if kind == "fan-out": + return [node["result"]["output"]["step_template"]] + if kind in {"while", "do-while"} and child is not node["children"][0]: + return steps_of(node["children"][0]) + return steps_of(child) + + def child_path(config, path, item): """Return a child's reporting path from its parent occurrence.""" return ( @@ -64,9 +81,12 @@ def child_path(config, path, item): ) -def walk_execution(seq, path=(), *, children_first=False, skip_done=False): +def walk_execution( + seq, path=(), *, children_first=False, skip_done=False, steps=None +): """Yield execution occurrences with one shared path rule.""" - for index, (config, node) in enumerate(zip(steps_of(seq), seq["nodes"])): + steps = steps_of(seq) if steps is None else steps + for index, (config, node) in enumerate(zip(steps, seq["nodes"])): if skip_done and node["phase"] == "done": continue here = [*path, config.get("id", f"step-{index}")] @@ -78,20 +98,31 @@ def walk_execution(seq, path=(), *, children_first=False, skip_done=False): child_path(config, here, item), children_first=True, skip_done=skip_done, + steps=child_steps(config, node, child), ) yield config, node, here if not children_first: for item, child in enumerate(children): - yield from walk_execution(child, child_path(config, here, item)) + yield from walk_execution( + child, + child_path(config, here, item), + steps=child_steps(config, node, child), + ) def validate_execution(tree: Any) -> None: """Validate stored structure without importing a project's custom steps.""" - def check_sequence(seq): - if not isinstance(seq, dict) or not isinstance(seq.get("source"), str): + def check_sequence(seq, steps=None, *, shared=False): + if not isinstance(seq, dict) or (shared and "source" in seq): raise ValueError("Invalid execution sequence") - steps = steps_of(seq) + if shared: + if not isinstance(steps, list): + raise ValueError("Invalid shared execution source") + else: + if not isinstance(seq.get("source"), str): + raise ValueError("Invalid execution sequence") + steps = steps_of(seq) nodes = seq.get("nodes") if ( not isinstance(steps, list) @@ -134,8 +165,6 @@ def check_sequence(seq): children = node.get("children", []) if not isinstance(children, list): raise ValueError("Invalid execution children") - for child in children: - check_sequence(child) binding = node.get("binding") if binding is not None: if step.get("type") != "workflow" or not isinstance(binding, dict): @@ -148,9 +177,21 @@ def check_sequence(seq): or not isinstance(binding.get("inputs"), dict) or len(children) != 1 or not isinstance(definition.get("steps"), list) - or definition["steps"] != steps_of(children[0]) ): raise ValueError("Invalid bound workflow definition or inputs") + for index, child in enumerate(children): + kind = step.get("type") + if kind == "workflow": + check_sequence(child, definition["steps"], shared=True) + elif kind == "fan-out": + template = (result or {}).get("output", {}).get("step_template") + if not isinstance(template, dict): + raise ValueError("Invalid fan-out template") + check_sequence(child, [template], shared=True) + elif kind in {"while", "do-while"} and index: + check_sequence(child, steps_of(children[0]), shared=True) + else: + check_sequence(child) if node["phase"] == "outputs" and binding is None: raise ValueError("Output finalization requires a workflow binding") if node["phase"] == "children" and ( @@ -272,7 +313,9 @@ def commit( qualified=qualified or name, ) for alias_context, alias, result in aliases: - self.project_alias(alias_context, alias, result) + self.project_alias( + alias_context, alias, node["result"] if result is None else result + ) self.state.save() def emit( @@ -305,8 +348,10 @@ def run( public=True, root=False, loop_alias=None, + steps=None, + aliases=(), ): - steps = steps_of(seq) + steps = steps_of(seq) if steps is None else steps for index, (config, node) in enumerate(zip(steps, seq["nodes"])): config = {"id": f"step-{index}", **config} name = config["id"] @@ -319,7 +364,14 @@ def run( ) self.state.current_step_id = name outcome = self.step( - config, node, context, ancestry, occurrence, public, qualified + config, + node, + context, + ancestry, + occurrence, + public, + qualified, + aliases=aliases, ) if outcome in HALTING: return outcome, node.get("error") @@ -345,6 +397,7 @@ def replay_fan_out(self, config, node, context, ancestry, path, public, qualifie path=(*path, "item", index), public=False, loop_alias=(qualified, index), + steps=child_steps(config, node, child), ) record = child["nodes"][0].get("result") if record is not None: @@ -357,7 +410,9 @@ def replay_fan_out(self, config, node, context, ancestry, path, public, qualifie return outcome, error return "completed", None - def step(self, config, node, context, ancestry, path, public, qualified): + def step( + self, config, node, context, ancestry, path, public, qualified, *, aliases=() + ): name = config.get("id", "step-0") kind = config.get("type", "command") if node["phase"] == "done": @@ -382,6 +437,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): loop_alias=(qualified, iteration) if kind in {"while", "do-while"} and iteration else None, + steps=child_steps(config, node, child), ) if outcome in HALTING: break @@ -391,7 +447,14 @@ def step(self, config, node, context, ancestry, path, public, qualified): if kind == "workflow": return self.workflow( - config, node, context, ancestry, path, public, qualified + config, + node, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) if node["phase"] in {"ready", "blocked"}: @@ -422,17 +485,26 @@ def step(self, config, node, context, ancestry, path, public, qualified): path, public, qualified, + aliases=aliases, ) result = impl.execute(config, context) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: return self.finish( - config, node, result, context, ancestry, path, public, qualified + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) children = [sequence(result.next_steps)] if result.next_steps else [] if kind == "fan-out": template = result.output.get("step_template", {}) children = ( - [sequence([template]) for _ in result.output.get("items", [])] + [occurrences([template]) for _ in result.output.get("items", [])] if template else [] ) @@ -441,7 +513,15 @@ def step(self, config, node, context, ancestry, path, public, qualified): if kind == "fan-out": result.output = {**result.output, "results": []} return self.finish( - config, node, result, context, ancestry, path, public, qualified + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) self.commit( node, @@ -472,14 +552,6 @@ def step(self, config, node, context, ancestry, path, public, qualified): **node["result"], "output": {**node["result"]["output"], "results": outputs}, } - self.commit( - node, - {"result": data}, - context=context, - name=name, - public=public, - qualified=qualified, - ) else: outcome, error = "completed", None for iteration, child in enumerate(node["children"]): @@ -493,6 +565,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): if kind in {"while", "do-while"} and iteration else None, + steps=child_steps(config, node, child), ) if outcome in HALTING: break @@ -503,7 +576,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): while len(node["children"]) < limit and evaluate_condition( config.get("condition", False), context ): - child = sequence(steps_of(node["children"][0])) + child = occurrences(steps_of(node["children"][0])) self.commit(node, {"children": [*node["children"], child]}) outcome, error = self.run( child, @@ -512,6 +585,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): path=(*path, len(node["children"]) - 1), public=public, loop_alias=(qualified, len(node["children"]) - 1), + steps=child_steps(config, node, child), ) if outcome in HALTING: break @@ -521,7 +595,13 @@ def step(self, config, node, context, ancestry, path, public, qualified): "phase": "done" if outcome == "completed" else "children", "outcome": outcome, "error": error, + **({"result": data} if kind == "fan-out" else {}), }, + context=context if kind == "fan-out" else None, + name=name if kind == "fan-out" else None, + public=public, + qualified=qualified, + aliases=aliases, ) return outcome @@ -560,7 +640,19 @@ def record(config, result, context): ) return data - def finish(self, config, node, result, context, ancestry, path, public, qualified): + def finish( + self, + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + *, + aliases=(), + ): name = config.get("id", "step-0") outcome = "completed" if result.status == StepStatus.PAUSED: @@ -581,6 +673,7 @@ def finish(self, config, node, result, context, ancestry, path, public, qualifie name=name, public=public, qualified=qualified, + aliases=aliases, ) self.emit( "step_completed", @@ -605,7 +698,18 @@ def finish(self, config, node, result, context, ancestry, path, public, qualifie ) return outcome - def workflow(self, config, node, context, ancestry, path, public, qualified): + def workflow( + self, + config, + node, + context, + ancestry, + path, + public, + qualified, + *, + aliases=(), + ): from .engine import WorkflowDefinition, workflow_dir_for self.emit( @@ -634,7 +738,7 @@ def workflow(self, config, node, context, ancestry, path, public, qualified): node, { "binding": binding, - "children": [sequence(definition.steps)], + "children": [occurrences(definition.steps)], "phase": "children", }, ) @@ -646,7 +750,15 @@ def workflow(self, config, node, context, ancestry, path, public, qualified): error=str(exc), ) return self.finish( - config, node, result, context, ancestry, path, public, qualified + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) else: definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) @@ -663,6 +775,7 @@ def workflow(self, config, node, context, ancestry, path, public, qualified): (*ancestry, target), path=(*path, "workflow"), public=False, + steps=child_steps(config, node, node["children"][0]), ) output = {"workflow": target, "status": outcome} if outcome == "completed": @@ -673,7 +786,15 @@ def workflow(self, config, node, context, ancestry, path, public, qualified): output.update(status="failed", error=str(exc)) result = StepResult(StepStatus.FAILED, output=output, error=str(exc)) return self.finish( - config, node, result, context, ancestry, path, public, qualified + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) elif outcome == "aborted": output["aborted"] = True @@ -694,7 +815,15 @@ def workflow(self, config, node, context, ancestry, path, public, qualified): ): config = {**config, "continue_on_error": False} return self.finish( - config, node, result, context, ancestry, path, public, qualified + config, + node, + result, + context, + ancestry, + path, + public, + qualified, + aliases=aliases, ) def fan_out(self, config, node, context, ancestry, path, public, qualified): @@ -707,6 +836,7 @@ def fan_out(self, config, node, context, ancestry, path, public, qualified): workers = min(workers, len(items)) initial = deepcopy(context.steps) halted = threading.Event() + template_name = output.get("step_template", {}).get("id", "item") def run_item(index): local = replace( @@ -720,21 +850,16 @@ def run_item(index): path=(*path, "item", index), public=False, loop_alias=(qualified, index), + steps=child_steps(config, node, child), + aliases=( + (context, f"{qualified}:{template_name}:{index}", None), + ) + if public + else (), ) if outcome in HALTING: halted.set() record = child["nodes"][0].get("result", {}) - template_name = output.get("step_template", {}).get("id", "item") - if public: - self.commit( - aliases=[ - ( - context, - f"{qualified}:{template_name}:{index}", - record, - ) - ] - ) return outcome, error, record.get("output", {}) results = [] diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 96f215a789..fff81a1b5e 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -1156,7 +1156,9 @@ def resume( raise ValueError(msg) if state.execution is not None: - persisted_steps = yaml.safe_load(state.execution["sequence"]["source"]) + from ._execution import steps_of + + persisted_steps = steps_of(state.execution["sequence"]) offset = state.execution.get("offset", 0) if persisted_steps != definition.steps[offset:]: raise ValueError("Invalid execution state: root sequence differs from workflow snapshot") @@ -1236,8 +1238,8 @@ def _execute_steps( if state.execution is None: state.execution = new_execution(steps, step_offset, context.steps) + state.save() context.steps = deepcopy(state.execution.get("initial", {})) - state.save() tree = state.execution["sequence"] executor = Execution(self, state, registry, rebind=rebind) outcome, error = executor.run(tree, context, (state.workflow_id,), root=True) diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 10079af431..f1c5014785 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -1516,6 +1516,215 @@ def test_native_yaml_definition_and_long_id_roundtrip(tmp_path, probe): assert WorkflowEngine(tmp_path).resume(state.run_id).status == RunStatus.PAUSED +def test_execution_shares_workflow_fan_out_and_loop_sources(tmp_path, probe): + child = definition( + "child", + [{"id": "work", "type": "probe"}], + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(), + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "item", "type": "probe"}, + }, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + ) + + tree = json.loads((state.runs_dir / "state.json").read_text())["execution"] + call_node, fan_node, loop_node, _ = tree["sequence"]["nodes"] + assert "source" not in call_node["children"][0] + assert "source" not in fan_node["children"][0] + assert "source" not in fan_node["children"][1] + assert "source" in loop_node["children"][0] + assert "source" not in loop_node["children"][1] + assert yaml.safe_load(call_node["binding"]["definition"])["steps"] == [ + {"id": "work", "type": "probe"} + ] + RunState.load(state.run_id, tmp_path) + + +@pytest.mark.parametrize( + "mutation", + [ + lambda tree: tree["sequence"]["nodes"][0]["children"][0].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][1]["children"][0].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][2]["children"][1].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][0]["children"][0]["nodes"].append( + {"phase": "ready"} + ), + ], +) +def test_shared_execution_sources_reject_local_copies(tmp_path, probe, mutation): + child = definition("child", [{"id": "work", "type": "probe"}]) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(), + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "item", "type": "probe"}, + }, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]) + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Invalid execution sequence"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert path.read_bytes() == before + + +@pytest.mark.parametrize("items", [1, 4]) +def test_fan_out_saves_once_per_item_transition(tmp_path, monkeypatch, probe, items): + original = RunState.save + saves = 0 + + def save(self): + nonlocal saves + saves += 1 + original(self) + + monkeypatch.setattr(RunState, "save", save) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": list(range(items)), + "max_concurrency": 1, + "step": {"id": "item", "type": "probe"}, + } + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert saves == items + 5 + + +def test_fan_out_snapshot_size_does_not_scale_with_template_length(tmp_path, probe): + def size(items, length): + root = tmp_path / f"fan-{items}-{length}" + state = WorkflowEngine(root).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": list(range(items)), + "max_concurrency": 1, + "step": { + "id": "item", + "type": "probe", + "payload": "x" * length, + }, + } + ], + ) + ) + return (state.runs_dir / "state.json").stat().st_size + + small_one, large_one = size(1, 128), size(1, 4096) + small_many, large_many = size(4, 128), size(4, 4096) + + assert large_many - small_many <= large_one - small_one + 256 + + +def test_loop_snapshot_size_does_not_scale_with_body_length(tmp_path, probe): + def size(iterations, length): + root = tmp_path / f"loop-{iterations}-{length}" + state = WorkflowEngine(root).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": iterations, + "steps": [ + { + "id": "body", + "type": "probe", + "payload": "x" * length, + } + ], + } + ], + ) + ) + return (state.runs_dir / "state.json").stat().st_size + + small_one, large_one = size(1, 128), size(1, 4096) + small_many, large_many = size(4, 128), size(4, 4096) + + assert large_many - small_many <= large_one - small_one + 256 + + +def test_tree_backed_resume_has_no_setup_checkpoint(tmp_path, monkeypatch, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + original = RunState.save + saves = 0 + + def save(self): + nonlocal saves + saves += 1 + original(self) + + monkeypatch.setattr(RunState, "save", save) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert saves == 3 + + def test_exact_depth_limit_is_allowed(tmp_path, probe): for index in range(1, 17): install( From 5565ecb387d7ee595411c7403d1fce22c567d124 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 19:58:34 +0200 Subject: [PATCH 08/28] docs(workflows): document composition resume Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- design/workflow-step.md | 19 ++++- docs/reference/workflows.md | 142 +++++++++++++++++++++++++++++------- workflows/ARCHITECTURE.md | 19 +++-- workflows/README.md | 6 +- 4 files changed, 150 insertions(+), 36 deletions(-) diff --git a/design/workflow-step.md b/design/workflow-step.md index ddb902d840..c03b0b1ec9 100644 --- a/design/workflow-step.md +++ b/design/workflow-step.md @@ -41,10 +41,21 @@ unless `continue_on_error: true` is set (an explicit abort always stops). The engine calls `validate()` during workflow validation but does not automatically validate a definition passed to `execute()`. Guard invalid configurations in `execute()` too, returning a failed result rather than a -successful default or an unhandled exception. New runs persist an execution -tree: resume retries unfinished occurrences and restores completed results and -selected expansions. Legacy states enter through their old top-level index. -Side effects performed before their completion checkpoint can still repeat. +successful default or an unhandled exception. + +New runs persist an execution tree. Resume replays completed occurrences into +their expression contexts without calling their implementations, then continues +at the unfinished occurrence. Selected branches, custom expansions, loop +iterations, fan-out items, and bound workflow definitions therefore remain +frozen across resume. This is intentionally forward-only: new resume inputs do +not reinterpret or repeat completed work. Legacy states enter the tree once +through their saved top-level index. + +Fan-out items have independent expression contexts. Their internal results do +not enter the shared parent context; the parent receives the qualified item +result and the fan-out step's ordered `output.results`. A side effect performed +before its completion checkpoint can still repeat after an interruption, so the +guarantee is at-least-once rather than exactly-once. The registry holds one shared instance per type. Concurrent `fan-out` can invoke that instance from multiple threads: keep execution stateless and diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 939791f5ab..46c25c4136 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -67,9 +67,17 @@ specify workflow resume | `-i` / `--input` | Updated input values as `key=value` (repeatable) | | `--json` | Emit the resume outcome as a single JSON object | -Resumes a paused or failed workflow run from the exact step where it stopped. Useful after responding to a gate step or fixing an issue that caused a failure. - -Supplied `--input` values are merged over the run's stored inputs and re-validated against the workflow's input types, then the blocked step is re-run with the updated values. This lets a run continue with information that only became available after it paused, or with a corrected value after a failure: +Resumes a paused or failed workflow run from its persisted execution state. A +`running` run is not resumable. Resume replays completed work into its +expression contexts without running step implementations again, then continues +at the unfinished occurrence. This lets a run continue with information that +only became available after it paused, or with a corrected value after a +failure. + +Supplied `--input` values are merged over the run's stored root inputs and +re-validated against the workflow's input types. Unknown root input names are +ignored. Updated values affect only unfinished work; completed steps and calls +are not reinterpreted or repeated: ```bash specify workflow resume --input cmd="exit 0" @@ -561,9 +569,11 @@ steps: ``` The included workflow sees only declared inputs passed through `input` and its -own step results. Unknown input names, missing required values, and invalid -types/enums fail the call. Existing defaults and `integration: auto` resolution -apply. Values cross back only through explicit declarations: +own step results. It does not inherit the caller's `inputs`, step results, +fan-out `item`/`fan_in` values, or workflow defaults. Unknown input names, +missing required values, and invalid types/enums fail the call. Existing child +defaults and `integration: auto` resolution apply. Values cross back only +through explicit declarations: ```yaml outputs: @@ -577,34 +587,114 @@ The caller reads `{{ steps.investigate.output.report }}`. Output includes are reserved. Returned values must be JSON-safe. Private inputs, step records, and logs are not part of the return value. No separate child run is created. -Failures may be handled with `continue_on_error: true` at either the included -step or workflow-call boundary. Pauses and explicit aborts always stop execution. +`continue_on_error: true` on a workflow call handles a returned child failure +and call-boundary contract failures, such as an unavailable target, invalid +mapped input, cycle, depth limit, or invalid declared output. It does not catch +step exceptions, expression errors (including `from_json` errors in `input:` or +`outputs:`), interruptions, or checkpoint failures; those propagate exactly as +they do at the root. An unavailable step implementation is terminal at every +boundary. Pauses and explicit aborts always stop execution. + Nested composition is allowed, but repeated workflow IDs on the active call path are cycles. Diamonds are allowed. The maximum included depth is 16, with the root at depth zero. +### Calling a Child Gate + +Only root-declared inputs can be supplied to `workflow resume --input`. Map a +root input through each workflow boundary when a child gate uses it as a +`verdict_input`: + +```yaml +# Parent workflow +inputs: + approval: + type: string + default: "" +steps: + - id: review-release + type: workflow + workflow: release-notes + input: + approval: "{{ inputs.approval }}" +``` + +```yaml +# Installed release-notes workflow +inputs: + approval: + type: string + default: "" +steps: + - id: review + type: gate + message: "Approve the release notes?" + options: [approve, reject] + verdict_input: approval +``` + +The initial run pauses if `approval` is empty. Its structured status identifies +the nested gate and its enclosing call with `gate.scope_path`. Resume the root +run, not the child, to continue it: + +```bash +specify workflow status --json +specify workflow resume --input approval=approve +``` + +A gate with `verdict_input` is not supported inside a fan-out item, including +through one or more workflow calls. The `inside_fan_out` runtime condition is +preserved across workflow boundaries. + +### Defaults and Errors at the Call Boundary + +Parent `integration`, `model`, and `options` defaults apply only to steps in +the parent workflow. A child uses its own defaults or automatic resolution. To +make a value common to both, declare it as a child input and map it explicitly. +The call result intentionally has an empty `input` field and does not expose +the child's private inputs, step records, or logs. + +An initial binding or output-finalization contract failure can be handled with +`continue_on_error`. A failure while rebinding an incomplete call during +`workflow resume --input` is different: it propagates, leaves the call and its +child subtree unchanged, and can be retried with corrected root inputs. + ### Execution identity and resume -Each step occurrence owns a record in a persisted execution tree. Authored step -names are local expression aliases, not global execution IDs. Fan-out items -have independent alias contexts; results remain ordered by item index. +Each step occurrence owns a record in a persisted execution tree. An authored +step ID is a local expression alias, not a global execution ID. Fan-out items +have independent alias contexts and ordered item results. Public reporting uses +qualified occurrence IDs where needed, for example `fan:template:0` for a +fan-out item and `loop:step:1` for a later loop iteration. Qualified IDs are +not expression names. New runs persist selected branches, dynamic custom-step expansions, loop -iterations, and fan-out items. Resume retries unfinished operations and retains -completed work without reevaluating already selected branches. Workflow targets -and overlay-resolved definitions remain bound even if installations change. -Ordinary resume retains bound inputs; explicit `--input` updates rebind reached, -incomplete calls through their original mappings. Completed calls retain their -results. Failed output evaluation retries finalization without repeating child -commands. +iterations, and fan-out items. Resume retains completed work without +reevaluating already selected branches. Workflow targets and overlay-resolved +definitions remain bound even if installations change. An unbound call still +checks that its target is installed and enabled when execution reaches it. -Snapshots are stored as YAML strings inside the private JSON execution tree, -preserving YAML scalar types. Inputs and results remain JSON values. Legacy runs -without a tree enter through their saved top-level index, then use tree-backed -resume. Resume is available only for runs in the `paused` or `failed` state. +Ordinary resume retains bound inputs. Explicit `--input` rebinds reached, +incomplete calls through their original mappings: newly mapped values override +the prior binding, while values not mapped again retain their bound values. +Completed calls retain their results. Failed output evaluation retries only +finalization, without repeating completed child commands. -Run/resume/status JSON includes `workflow_scopes` summaries when calls exist and -reports the active nested gate with its `scope_path`. +Snapshots are stored as YAML strings inside the private JSON execution tree, +preserving YAML scalar types. Inputs and results remain JSON values. The +snapshot freezes workflow definitions and expansions, not files under +`context.workflow_dir` or step implementations; missing resources can still +cause ordinary step failures. Legacy runs without a tree enter through their +saved root index, then use tree-backed resume. Resume is available only for +runs in the `paused` or `failed` state. + +`current_step_index` is the root-sequence index; `current_step_id` is the local +ID of the active leaf. Structured run/status output includes `workflow_scopes` +summaries when calls exist and reports an active nested gate with its +`scope_path`. Private log events add `workflow_id` and `execution_path` to the +qualified `step_id`. A step emits completion after its checkpoint; containers +emit before their children, while replayed completed occurrences emit no step +events. > **Security note:** a `shell` step runs a local command with **your** privileges. There is no capability sandbox — `requires` is an advisory pre-condition block (spec-kit version, integrations), not a runtime gate, so it does **not** restrict what a step can do. In particular there is no `requires.permissions` capability gate: it is rejected by validation precisely because it would imply a sandbox that does not exist. Review any catalog or downloaded workflow before running it, and use a `gate` step to require explicit approval before sensitive or destructive shell commands. @@ -718,7 +808,9 @@ Each workflow run persists its state at `.specify/workflows/runs//`: - `inputs.json` — resolved input values - `log.jsonl` — step-by-step execution log -This enables `specify workflow resume` to continue from the exact step where a run was paused (e.g., at a gate) or failed. +This enables `specify workflow resume` to replay completed occurrences and +continue at the unfinished occurrence where a run paused (for example, at a +gate) or failed. ### Gate Verdict Inputs diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index a9ee91c00d..8e5397fc2a 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -70,14 +70,21 @@ flowchart LR E -- "resume()" --> B ``` -When a `gate` step pauses execution, the engine persists `current_step_index` and all accumulated `step_results`. On `specify workflow resume `, the engine restores the context and continues from the paused step. +When a `gate` step pauses execution, the engine persists `current_step_index` +and all accumulated `step_results`. On `specify workflow resume `, the +same executor replays completed occurrences into their contexts without +executing them, then continues at the unfinished occurrence. New runs use a versioned execution tree. Each occurrence owns its result, -selected child sequences, and optional workflow binding. The same executor -traverses that tree on initial execution and resume, restoring local aliases -from completed results. Fan-out items have separate contexts. Legacy runs enter -through their top-level index once. Inputs and tree transitions share one atomic -state checkpoint; the inputs file is a compatibility mirror. +selected child sequences, and optional workflow binding. A binding stores the +target, frozen definition, private inputs, and `workflow_dir`; the called +workflow remains a private scope in the same run. Fan-out items have separate +contexts. Binding and selected expansions are checkpointed before child side +effects, results before an occurrence is done, and logs after the checkpoint. +Legacy runs enter through their top-level index once. Inputs and tree +transitions share one atomic state checkpoint; the inputs file is a +compatibility mirror. A checkpoint failure prevents further writes by that +executor instance. ## Step Types diff --git a/workflows/README.md b/workflows/README.md index 28ac151288..b343c9ac4a 100644 --- a/workflows/README.md +++ b/workflows/README.md @@ -4,7 +4,11 @@ Workflows are multi-step, resumable automation pipelines defined in YAML. They o ## How It Works -A workflow definition declares a sequence of steps. The engine executes them in order, dispatching commands to AI integrations, running shell commands, evaluating conditions for branching, and pausing at gates for human review. State is persisted after each step, so workflows can be resumed after interruption. +A workflow definition declares a sequence of steps. The engine executes them in +order, dispatching commands to AI integrations, running shell commands, +evaluating conditions for branching, and pausing at gates for human review. +Persisted execution transitions let workflows resume after interruption without +re-running completed work. ```yaml steps: From fa0dda8e162b0022cf8ce1874ef9eaa7cc48ba57 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 20:11:44 +0200 Subject: [PATCH 09/28] fix(workflows): preserve replay projections Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- docs/reference/workflows.md | 9 +- src/specify_cli/workflows/_execution.py | 8 +- tests/workflows/test_composition_execution.py | 117 +++++++++++++++++- 3 files changed, 128 insertions(+), 6 deletions(-) diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 46c25c4136..cb540a7830 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -582,10 +582,11 @@ outputs: ``` The caller reads `{{ steps.investigate.output.report }}`. Output includes -`workflow` and `status`; failures also include `error`, and an abort includes -`aborted: true`. These names and `integration`, `model`, `options`, and `input` -are reserved. Returned values must be JSON-safe. Private inputs, step records, -and logs are not part of the return value. No separate child run is created. +`workflow` and `status`; failures include `error` when the failed operation +reported one, and an abort includes `aborted: true`. These names and +`integration`, `model`, `options`, and `input` are reserved. Returned values +must be JSON-safe. Private inputs, step records, and logs are not part of the +return value. No separate child run is created. `continue_on_error: true` on a workflow call handles a returned child failure and call-boundary contract failures, such as an unavailable target, invalid diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index db64d843f6..f5e5168ea4 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -175,6 +175,8 @@ def check_sequence(seq, steps=None, *, shared=False): or not isinstance(definition.get("workflow"), dict) or definition["workflow"].get("id") != binding.get("workflow") or not isinstance(binding.get("inputs"), dict) + or "workflow_dir" not in binding + or not isinstance(binding["workflow_dir"], (str, type(None))) or len(children) != 1 or not isinstance(definition.get("steps"), list) ): @@ -400,7 +402,7 @@ def replay_fan_out(self, config, node, context, ancestry, path, public, qualifie steps=child_steps(config, node, child), ) record = child["nodes"][0].get("result") - if record is not None: + if public and record is not None: self.project_alias( context, f"{qualified}:{template_name}:{index}", @@ -419,6 +421,10 @@ def step( self.project( context, name, node["result"], public=public, qualified=qualified ) + for alias_context, alias, result in aliases: + self.project_alias( + alias_context, alias, node["result"] if result is None else result + ) if kind == "workflow": return node.get("outcome", "completed") if kind == "fan-out": diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index f1c5014785..066d1e299c 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -461,11 +461,30 @@ def test_legacy_resume_adapts_once(tmp_path, probe): lambda tree: tree.update(version=99), lambda tree: tree["sequence"].update(nodes=[]), lambda tree: tree["sequence"]["nodes"][0].update(phase="nonsense"), + lambda tree: tree["sequence"]["nodes"][0]["binding"].pop("workflow_dir"), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + workflow_dir=42 + ), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + workflow_dir={} + ), ], ) def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) state = WorkflowEngine(tmp_path).execute( - definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) ) path = state.runs_dir / "state.json" data = json.loads(path.read_text()) @@ -477,6 +496,32 @@ def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): assert path.read_bytes() == before +def test_bound_call_without_source_path_resumes_with_null_workflow_dir( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + child = definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + monkeypatch.setattr(execution, "resolve_target", lambda *_: child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + assert state.execution["sequence"]["nodes"][0]["binding"]["workflow_dir"] is None + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + + @pytest.mark.parametrize("failure_after_replace", [False, True]) def test_checkpoint_failure_leaves_running_run_not_resumable( tmp_path, monkeypatch, probe, failure_after_replace @@ -647,6 +692,76 @@ def test_replay_restores_fan_out_aliases_from_completed_if(tmp_path, probe): assert state.step_results["join"]["output"]["results"] == [{"value": 1}] +def test_resume_restores_completed_fan_out_item_aliases(tmp_path, monkeypatch, probe): + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + if context.item == 2 and not context.inputs["approve"]: + return StepResult(StepStatus.PAUSED) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "template", "type": "pause-second"}, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan:template:0", "fan:template:1"], + }, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["join"]["output"]["results"] == [ + {"value": 1}, + {"value": 2}, + ] + + +def test_replay_keeps_private_nested_fan_out_aliases_private(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "outer", + "type": "fan-out", + "items": [1], + "step": { + "id": "outer-item", + "type": "fan-out", + "items": [1], + "step": {"id": "inner", "type": "probe"}, + }, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert "outer:outer-item:0" in state.step_results + assert "outer:outer-item:0:inner:0" not in state.step_results + + def test_replay_does_not_execute_completed_fan_out_or_emit_callbacks(tmp_path, probe): callbacks = [] engine = WorkflowEngine(tmp_path) From 1a3ca48679fd0df9c89c42b9a4fdbb124d49dab1 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 20:42:53 +0200 Subject: [PATCH 10/28] fix(workflows): reject inconsistent execution trees Assisted-by: OpenCode (model: deepseek-v4.1-flash, autonomous) --- src/specify_cli/workflows/_execution.py | 91 ++++++++++--------- tests/workflows/test_composition_execution.py | 58 ++++++++++++ 2 files changed, 107 insertions(+), 42 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index f5e5168ea4..61dabd11d8 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -60,23 +60,31 @@ def steps_of(seq: dict[str, Any]) -> Any: return yaml.safe_load(seq["source"]) -def child_steps(config, node, child): +LOOP_TYPES = frozenset({"while", "do-while"}) + + +def shares_source(kind, index): + """Whether child ``index`` reads its steps from the parent occurrence.""" + return kind in {"workflow", "fan-out"} or (kind in LOOP_TYPES and index > 0) + + +def child_steps(config, node, index): """Read a child sequence from its position in the parent occurrence.""" kind = config.get("type") + if not shares_source(kind, index): + return steps_of(node["children"][index]) if kind == "workflow": return yaml.safe_load(node["binding"]["definition"])["steps"] if kind == "fan-out": return [node["result"]["output"]["step_template"]] - if kind in {"while", "do-while"} and child is not node["children"][0]: - return steps_of(node["children"][0]) - return steps_of(child) + return steps_of(node["children"][0]) def child_path(config, path, item): """Return a child's reporting path from its parent occurrence.""" return ( [*path, str(item)] - if config.get("type") in {"fan-out", "while", "do-while"} + if config.get("type") in {"fan-out"} | LOOP_TYPES else path ) @@ -90,24 +98,18 @@ def walk_execution( if skip_done and node["phase"] == "done": continue here = [*path, config.get("id", f"step-{index}")] - children = node.get("children", []) - if children_first: - for item, child in enumerate(children): - yield from walk_execution( - child, - child_path(config, here, item), - children_first=True, - skip_done=skip_done, - steps=child_steps(config, node, child), - ) - yield config, node, here if not children_first: - for item, child in enumerate(children): - yield from walk_execution( - child, - child_path(config, here, item), - steps=child_steps(config, node, child), - ) + yield config, node, here + for item, child in enumerate(node.get("children", [])): + yield from walk_execution( + child, + child_path(config, here, item), + children_first=children_first, + skip_done=skip_done, + steps=child_steps(config, node, item), + ) + if children_first: + yield config, node, here def validate_execution(tree: Any) -> None: @@ -165,9 +167,10 @@ def check_sequence(seq, steps=None, *, shared=False): children = node.get("children", []) if not isinstance(children, list): raise ValueError("Invalid execution children") + kind = step.get("type") binding = node.get("binding") if binding is not None: - if step.get("type") != "workflow" or not isinstance(binding, dict): + if kind != "workflow" or not isinstance(binding, dict): raise ValueError("Invalid workflow binding") definition = yaml.safe_load(binding.get("definition", "")) if ( @@ -181,19 +184,23 @@ def check_sequence(seq, steps=None, *, shared=False): or not isinstance(definition.get("steps"), list) ): raise ValueError("Invalid bound workflow definition or inputs") + elif kind == "workflow" and children: + raise ValueError("Workflow children require a binding") + if kind == "fan-out" and children: + template = (result or {}).get("output", {}).get("step_template") + if not isinstance(template, dict): + raise ValueError("Invalid fan-out template") for index, child in enumerate(children): - kind = step.get("type") - if kind == "workflow": - check_sequence(child, definition["steps"], shared=True) - elif kind == "fan-out": - template = (result or {}).get("output", {}).get("step_template") - if not isinstance(template, dict): - raise ValueError("Invalid fan-out template") - check_sequence(child, [template], shared=True) - elif kind in {"while", "do-while"} and index: - check_sequence(child, steps_of(children[0]), shared=True) + if shares_source(kind, index): + check_sequence(child, child_steps(step, node, index), shared=True) else: check_sequence(child) + if ( + node["phase"] == "done" + and kind != "workflow" + and any(nested["phase"] != "done" for nested in child["nodes"]) + ): + raise ValueError("Completed execution has unfinished children") if node["phase"] == "outputs" and binding is None: raise ValueError("Output finalization requires a workflow binding") if node["phase"] == "children" and ( @@ -399,7 +406,7 @@ def replay_fan_out(self, config, node, context, ancestry, path, public, qualifie path=(*path, "item", index), public=False, loop_alias=(qualified, index), - steps=child_steps(config, node, child), + steps=child_steps(config, node, index), ) record = child["nodes"][0].get("result") if public and record is not None: @@ -441,9 +448,9 @@ def step( path=(*path, iteration), public=public, loop_alias=(qualified, iteration) - if kind in {"while", "do-while"} and iteration + if kind in LOOP_TYPES and iteration else None, - steps=child_steps(config, node, child), + steps=child_steps(config, node, iteration), ) if outcome in HALTING: break @@ -568,14 +575,14 @@ def step( path=(*path, iteration), public=public, loop_alias=(qualified, iteration) - if kind in {"while", "do-while"} + if kind in LOOP_TYPES and iteration else None, - steps=child_steps(config, node, child), + steps=child_steps(config, node, iteration), ) if outcome in HALTING: break - if outcome == "completed" and kind in {"while", "do-while"}: + if outcome == "completed" and kind in LOOP_TYPES: limit = config.get("max_iterations", 10) if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1: limit = 10 @@ -591,7 +598,7 @@ def step( path=(*path, len(node["children"]) - 1), public=public, loop_alias=(qualified, len(node["children"]) - 1), - steps=child_steps(config, node, child), + steps=child_steps(config, node, len(node["children"]) - 1), ) if outcome in HALTING: break @@ -781,7 +788,7 @@ def workflow( (*ancestry, target), path=(*path, "workflow"), public=False, - steps=child_steps(config, node, node["children"][0]), + steps=child_steps(config, node, 0), ) output = {"workflow": target, "status": outcome} if outcome == "completed": @@ -856,7 +863,7 @@ def run_item(index): path=(*path, "item", index), public=False, loop_alias=(qualified, index), - steps=child_steps(config, node, child), + steps=child_steps(config, node, index), aliases=( (context, f"{qualified}:{template_name}:{index}", None), ) diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 066d1e299c..99da96329c 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -496,6 +496,64 @@ def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): assert path.read_bytes() == before +@pytest.mark.parametrize("index", [0, 1]) +def test_unbound_workflow_children_rejected_without_writes(tmp_path, probe, index): + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(id="first"), + call(id="second"), + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][index].pop("binding") + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + + +def test_completed_container_with_unfinished_child_rejected(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [{"id": "work", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][0]["children"][0]["nodes"][0] = { + "phase": "ready" + } + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + assert probe["work"] == 1 + + def test_bound_call_without_source_path_resumes_with_null_workflow_dir( tmp_path, monkeypatch, probe ): From 612cdf372e1f2e1ba930d564fd2da986a45f607d Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 20:44:23 +0200 Subject: [PATCH 11/28] fix(workflows): narrow workflow target resolution failures Assisted-by: OpenCode (model: deepseek-v4.1-flash, autonomous) --- src/specify_cli/workflows/_execution.py | 22 +++++--------- src/specify_cli/workflows/composition.py | 2 +- tests/workflows/test_composition_execution.py | 30 +++++++++++++++++++ 3 files changed, 38 insertions(+), 16 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 61dabd11d8..859047e129 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -621,31 +621,23 @@ def step( @staticmethod def record(config, result, context): output = result.output - is_workflow = config.get("type") == "workflow" data = { "type": config.get("type", "command"), - "integration": None - if is_workflow - else output.get("integration") + "integration": output.get("integration") or config.get("integration") or context.default_integration, - "model": None - if is_workflow - else output.get("model") + "model": output.get("model") or config.get("model") or context.default_model, - "options": {} - if is_workflow - else output.get("options") or config.get("options", {}), - "input": ( - {} - if is_workflow - else output.get("input") or config.get("input", {}) - ), + "options": output.get("options") or config.get("options", {}), + "input": output.get("input") or config.get("input", {}), "output": output, "status": result.status.value, "error": result.error, } + if config.get("type") == "workflow": + # A call result exposes neither caller defaults nor private inputs (4.3). + data.update(integration=None, model=None, options={}, input={}) if data["type"] == "command" and "integration_args" in output: data.update( integration_args=output["integration_args"], diff --git a/src/specify_cli/workflows/composition.py b/src/specify_cli/workflows/composition.py index 51db44e0c4..de0de3a355 100644 --- a/src/specify_cli/workflows/composition.py +++ b/src/specify_cli/workflows/composition.py @@ -127,7 +127,7 @@ def resolve_target(project_root: Path, target: Any, ancestry: tuple[str, ...]): return definition except CallError: raise - except Exception as exc: + except (OSError, ValueError) as exc: raise CallError(str(exc)) from exc diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 99da96329c..519bf5eeb5 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -1220,6 +1220,36 @@ def test_resolution_failures_are_call_failures(tmp_path, probe, mode): assert not probe +@pytest.mark.parametrize( + "error, handled", + [ + (ValueError("bad overlay"), True), + (FileNotFoundError("gone"), True), + (RuntimeError("resolver bug"), False), + ], +) +def test_only_resolver_contract_errors_are_call_failures( + tmp_path, monkeypatch, probe, error, handled +): + from specify_cli.workflows.overlay.resolver import WorkflowResolver + + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + + def resolve(self, workflow_id): + raise error + + monkeypatch.setattr(WorkflowResolver, "resolve", resolve) + parent = definition("parent", [call(continue_on_error=True)]) + if handled: + state = WorkflowEngine(tmp_path).execute(parent) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + else: + with pytest.raises(RuntimeError, match="resolver bug"): + WorkflowEngine(tmp_path).execute(parent) + assert not probe + + @pytest.mark.parametrize( "child_continue, call_continue", [(False, False), (True, False), (False, True)] ) From 9fe722ac56b42a1b1f170838b5a8829efe1485e0 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 21:18:57 +0200 Subject: [PATCH 12/28] refactor(workflows): replay fan-out through the live path Assisted-by: OpenCode (model: deepseek-v4.1-flash, autonomous) --- src/specify_cli/workflows/_execution.py | 146 ++++++------------ tests/workflows/test_composition_execution.py | 95 ++++++++++++ 2 files changed, 143 insertions(+), 98 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 859047e129..7c073f720e 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -271,12 +271,17 @@ def child_context_for_call(parent, binding, definition): def qualified_id(name, loop_alias=None): - """Return the public occurrence ID for a direct loop body step.""" + """Return the public occurrence ID of a loop-iteration or fan-out-item step.""" if loop_alias is None: return name return f"{loop_alias[0]}:{name}:{loop_alias[1]}" +def loop_alias_for(kind, qualified, iteration): + """Qualify direct body steps of later loop iterations (4.6).""" + return (qualified, iteration) if kind in LOOP_TYPES and iteration else None + + class Execution: def __init__(self, engine, state, registry, *, rebind=False): self.engine, self.state, self.registry = engine, state, registry @@ -305,7 +310,6 @@ def commit( name=None, public=False, qualified=None, - aliases=(), ): """Mutate an occurrence and its compatibility views in one checkpoint.""" with self.state._lock: @@ -321,10 +325,6 @@ def commit( public=public, qualified=qualified or name, ) - for alias_context, alias, result in aliases: - self.project_alias( - alias_context, alias, node["result"] if result is None else result - ) self.state.save() def emit( @@ -358,7 +358,6 @@ def run( root=False, loop_alias=None, steps=None, - aliases=(), ): steps = steps_of(seq) if steps is None else steps for index, (config, node) in enumerate(zip(steps, seq["nodes"])): @@ -380,81 +379,51 @@ def run( occurrence, public, qualified, - aliases=aliases, ) if outcome in HALTING: return outcome, node.get("error") return "completed", None - def replay_fan_out(self, config, node, context, ancestry, path, public, qualified): - """Replay persisted fan-out items in order without starting workers.""" - output = node["result"]["output"] - initial = deepcopy(context.steps) - items = output.get("items", []) - template_name = output.get("step_template", {}).get("id", "item") - for index, child in enumerate(node.get("children", [])): - local = replace( - context, - steps=deepcopy(initial), - item=items[index], - inside_fan_out=True, - ) + def run_children(self, config, node, context, ancestry, path, public, qualified): + """Run or replay persisted child sequences in order.""" + kind = config.get("type", "command") + for iteration, child in enumerate(node.get("children", [])): outcome, error = self.run( child, - local, + context, ancestry, - path=(*path, "item", index), - public=False, - loop_alias=(qualified, index), - steps=child_steps(config, node, index), + path=(*path, iteration), + public=public, + loop_alias=loop_alias_for(kind, qualified, iteration), + steps=child_steps(config, node, iteration), ) - record = child["nodes"][0].get("result") - if public and record is not None: - self.project_alias( - context, - f"{qualified}:{template_name}:{index}", - record, - ) if outcome in HALTING: return outcome, error return "completed", None - def step( - self, config, node, context, ancestry, path, public, qualified, *, aliases=() - ): + def step(self, config, node, context, ancestry, path, public, qualified): name = config.get("id", "step-0") kind = config.get("type", "command") if node["phase"] == "done": self.project( context, name, node["result"], public=public, qualified=qualified ) - for alias_context, alias, result in aliases: - self.project_alias( - alias_context, alias, node["result"] if result is None else result + if kind == "fan-out" and node.get("children"): + self.fan_out( + config, + node, + context, + ancestry, + path, + public, + qualified, + sequential=True, ) - if kind == "workflow": - return node.get("outcome", "completed") - if kind == "fan-out": - outcome, _ = self.replay_fan_out( + elif kind not in {"workflow", "fan-out"}: + self.run_children( config, node, context, ancestry, path, public, qualified ) - else: - outcome = "completed" - for iteration, child in enumerate(node.get("children", [])): - outcome, _ = self.run( - child, - context, - ancestry, - path=(*path, iteration), - public=public, - loop_alias=(qualified, iteration) - if kind in LOOP_TYPES and iteration - else None, - steps=child_steps(config, node, iteration), - ) - if outcome in HALTING: - break - return outcome if outcome in HALTING else node.get("outcome", "completed") + return "completed" if node.get("outcome") == "aborted": return "aborted" @@ -467,7 +436,6 @@ def step( path, public, qualified, - aliases=aliases, ) if node["phase"] in {"ready", "blocked"}: @@ -498,7 +466,6 @@ def step( path, public, qualified, - aliases=aliases, ) result = impl.execute(config, context) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: @@ -511,7 +478,6 @@ def step( path, public, qualified, - aliases=aliases, ) children = [sequence(result.next_steps)] if result.next_steps else [] if kind == "fan-out": @@ -534,7 +500,6 @@ def step( path, public, qualified, - aliases=aliases, ) self.commit( node, @@ -566,22 +531,9 @@ def step( "output": {**node["result"]["output"], "results": outputs}, } else: - outcome, error = "completed", None - for iteration, child in enumerate(node["children"]): - outcome, error = self.run( - child, - context, - ancestry, - path=(*path, iteration), - public=public, - loop_alias=(qualified, iteration) - if kind in LOOP_TYPES - and iteration - else None, - steps=child_steps(config, node, iteration), - ) - if outcome in HALTING: - break + outcome, error = self.run_children( + config, node, context, ancestry, path, public, qualified + ) if outcome == "completed" and kind in LOOP_TYPES: limit = config.get("max_iterations", 10) if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1: @@ -597,7 +549,9 @@ def step( ancestry, path=(*path, len(node["children"]) - 1), public=public, - loop_alias=(qualified, len(node["children"]) - 1), + loop_alias=loop_alias_for( + kind, qualified, len(node["children"]) - 1 + ), steps=child_steps(config, node, len(node["children"]) - 1), ) if outcome in HALTING: @@ -614,7 +568,6 @@ def step( name=name if kind == "fan-out" else None, public=public, qualified=qualified, - aliases=aliases, ) return outcome @@ -655,8 +608,6 @@ def finish( path, public, qualified, - *, - aliases=(), ): name = config.get("id", "step-0") outcome = "completed" @@ -678,7 +629,6 @@ def finish( name=name, public=public, qualified=qualified, - aliases=aliases, ) self.emit( "step_completed", @@ -712,8 +662,6 @@ def workflow( path, public, qualified, - *, - aliases=(), ): from .engine import WorkflowDefinition, workflow_dir_for @@ -763,7 +711,6 @@ def workflow( path, public, qualified, - aliases=aliases, ) else: definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) @@ -799,7 +746,6 @@ def workflow( path, public, qualified, - aliases=aliases, ) elif outcome == "aborted": output["aborted"] = True @@ -828,16 +774,19 @@ def workflow( path, public, qualified, - aliases=aliases, ) - def fan_out(self, config, node, context, ancestry, path, public, qualified): + def fan_out( + self, config, node, context, ancestry, path, public, qualified, *, sequential=False + ): output = node["result"]["output"] items = output.get("items", []) try: workers = max(1, int(output.get("max_concurrency", 1))) except (TypeError, ValueError, OverflowError): workers = 1 + if sequential: + workers = 1 workers = min(workers, len(items)) initial = deepcopy(context.steps) halted = threading.Event() @@ -856,16 +805,17 @@ def run_item(index): public=False, loop_alias=(qualified, index), steps=child_steps(config, node, index), - aliases=( - (context, f"{qualified}:{template_name}:{index}", None), - ) - if public - else (), ) if outcome in HALTING: halted.set() - record = child["nodes"][0].get("result", {}) - return outcome, error, record.get("output", {}) + record = child["nodes"][0].get("result") + if public and record is not None: + # Projection only (E2): persisted by the next commit, rebuilt on replay. + with self.state._lock: + self.project_alias( + context, f"{qualified}:{template_name}:{index}", record + ) + return outcome, error, (record or {}).get("output", {}) results = [] if workers <= 1: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 519bf5eeb5..54f654329d 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -858,6 +858,101 @@ def test_replay_does_not_execute_completed_fan_out_or_emit_callbacks(tmp_path, p ] == [("step_started", "wait"), ("step_completed", "wait")] +def test_fan_out_item_aliases_are_projected_under_run_lock(tmp_path, monkeypatch): + from specify_cli.workflows._execution import Execution + + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + if context.item == 2 and not context.inputs["approve"]: + return StepResult(StepStatus.PAUSED) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"id": "template", "type": "pause-second"}, + } + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + owned = [] + original = Execution.project_alias + + def spy(self, context, name, result): + owned.append(self.state._lock._is_owned()) + return original(self, context, name, result) + + monkeypatch.setattr(Execution, "project_alias", spy) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert owned and all(owned) + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["fan:template:1"]["output"] == {"value": 2} + + +def test_replay_does_not_start_worker_threads(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"id": "template", "type": "probe"}, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + class NoThreads: + def __init__(self, *args, **kwargs): + raise AssertionError("replay must not start worker threads") + + monkeypatch.setattr(execution, "ThreadPoolExecutor", NoThreads) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"template": 2, "wait": 2} + + +def test_fan_out_without_children_survives_replay(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "fan", "type": "fan-out", "items": [1, 2], "step": {}}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan"]["output"]["results"] == [] + + def test_replay_does_not_evaluate_completed_loop_condition( tmp_path, monkeypatch, probe ): From cd911dcd79006d65d29ccba558ae4c685444e955 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 21:23:03 +0200 Subject: [PATCH 13/28] fix(workflows): match main events for unknown step types Assisted-by: OpenCode (model: deepseek-v4.1-flash, autonomous) --- src/specify_cli/workflows/_execution.py | 63 ++++++++++--------- tests/test_workflows.py | 25 ++++++++ tests/workflows/test_composition_execution.py | 28 +++++++++ 3 files changed, 86 insertions(+), 30 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 7c073f720e..6f80e38a2e 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -26,6 +26,17 @@ HALTING = {"paused", "failed", "aborted"} +UNKNOWN_STEP_PREFIX = "Unknown step type: " + + +def unknown_step_error(kind): + return f"{UNKNOWN_STEP_PREFIX}{kind!r}" + + +def is_unknown_step_error(error): + """A missing implementation stays terminal across workflow boundaries.""" + return isinstance(error, str) and error.startswith(UNKNOWN_STEP_PREFIX) + class CheckpointError(RuntimeError): """Persistence failed; reload the authoritative disk checkpoint before retrying.""" @@ -332,16 +343,15 @@ def emit( event, qualified, path, - workflow, + ancestry, *, - private=False, callback_label=None, **fields, ): """Emit one step event and its optional start callback.""" entry = {"event": event, "step_id": qualified, **fields} - if private: - entry.update(execution_path=list(path), workflow_id=workflow) + if len(ancestry) > 1: + entry.update(execution_path=list(path), workflow_id=ancestry[-1]) self.state.append_log(entry) if callback_label is not None and self.engine.on_step_start is not None: with self.engine._callback_lock: @@ -445,28 +455,27 @@ def step(self, config, node, context, ancestry, path, public, qualified): "step_started", qualified, path, - ancestry[-1], - private=len(ancestry) > 1, + ancestry, type=kind, callback_label=config.get("command", "") or kind, ) impl = self.registry.get(kind) if impl is None: - # A missing implementation is a runtime configuration error, not - # a step failure that a workflow may explicitly recover from. - result = StepResult( - StepStatus.FAILED, error=f"Unknown step type: {kind!r}" - ) - return self.finish( - {**config, "continue_on_error": False}, + # As on main: terminal, only step_failed, no projected result. + # The node keeps its result so resume can retry after reinstalling. + error = unknown_step_error(kind) + result = StepResult(StepStatus.FAILED, error=error) + self.commit( node, - result, - context, - ancestry, - path, - public, - qualified, + { + "phase": "blocked", + "result": self.record(config, result, context), + "outcome": "failed", + "error": error, + }, ) + self.emit("step_failed", qualified, path, ancestry, error=error) + return "failed" result = impl.execute(config, context) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: return self.finish( @@ -513,8 +522,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): "step_completed", qualified, path, - ancestry[-1], - private=len(ancestry) > 1, + ancestry, status=result.status.value, ) else: @@ -634,8 +642,7 @@ def finish( "step_completed", qualified, path, - ancestry[-1], - private=len(ancestry) > 1, + ancestry, status=result.status.value, ) if result.status == StepStatus.FAILED: @@ -647,8 +654,7 @@ def finish( event, qualified, path, - ancestry[-1], - private=len(ancestry) > 1, + ancestry, error=result.error, ) return outcome @@ -669,8 +675,7 @@ def workflow( "step_started", qualified, path, - ancestry[-1], - private=len(ancestry) > 1, + ancestry, type="workflow", callback_label="workflow", ) @@ -761,9 +766,7 @@ def workflow( result = StepResult(status, output=output, error=error) # An unavailable child implementation is terminal at every workflow # boundary, matching direct execution of the same step. - if isinstance(result.error, str) and result.error.startswith( - "Unknown step type: " - ): + if is_unknown_step_error(result.error): config = {**config, "continue_on_error": False} return self.finish( config, diff --git a/tests/test_workflows.py b/tests/test_workflows.py index cd10c744bb..98b45f1e39 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -7770,6 +7770,31 @@ def test_unknown_step_type_ignores_continue_on_error(self, project_dir): assert "step_failed" in events assert "step_continue_on_error" not in events + def test_unknown_step_type_events_match_main(self, project_dir): + """Only step_started and step_failed, and no projected step result.""" + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + from specify_cli.workflows.base import RunStatus + + definition = WorkflowDefinition.from_string(""" +schema_version: "1.0" +workflow: + id: "unknown-type-events" + name: "Unknown Type Events" + version: "1.0.0" +steps: + - id: mystery + type: definitely-not-a-real-step +""") + state = WorkflowEngine(project_dir).execute(definition) + + assert state.status == RunStatus.FAILED + assert [ + entry["event"] + for entry in state.log_entries + if entry.get("step_id") == "mystery" + ] == ["step_started", "step_failed"] + assert "mystery" not in state.step_results + # ===== State Persistence Tests ===== diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 54f654329d..312addcac3 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -1396,6 +1396,34 @@ def execute(self, config, context): events = [entry["event"] for entry in state.log_entries] assert "step_failed" in events assert "step_continue_on_error" not in events + assert [ + entry["event"] + for entry in state.log_entries + if entry.get("step_id") == "missing" and entry.get("workflow_id") == "child" + ] == ["step_started", "step_failed"] + + +def test_unknown_step_type_resumes_after_reinstall(tmp_path, monkeypatch, probe): + class Reinstalled(StepBase): + type_key = "temporarily-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "missing", "type": "temporarily-installed"}], + ), + run_id="reinstall", + ) + assert state.status == RunStatus.FAILED + + monkeypatch.setitem(STEP_REGISTRY, "temporarily-installed", Reinstalled()) + state = WorkflowEngine(tmp_path).resume("reinstall") + + assert state.status == RunStatus.COMPLETED + assert state.step_results["missing"]["status"] == "completed" def test_unknown_fan_out_template_step_always_fails_despite_continue_on_error( From 6792bea1aaff941f1767a5c632da84d72f183337 Mon Sep 17 00:00:00 2001 From: Markus Date: Sun, 27 Sep 2026 21:53:30 +0200 Subject: [PATCH 14/28] fix(workflows): omit unprojected fan-out item results Preserve the item traversal's projection decision for missing step types. Cover sequential and parallel execution, inherited aliases, unnamed templates, and resume after reinstalling the implementation. Assisted-by: OpenCode (model: gpt-6-astra, autonomous) --- src/specify_cli/workflows/_execution.py | 6 +++ tests/workflows/test_composition_execution.py | 46 +++++++++++++++++-- 2 files changed, 48 insertions(+), 4 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 6f80e38a2e..e87e487b9a 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -799,6 +799,8 @@ def run_item(index): local = replace( context, steps=deepcopy(initial), item=items[index], inside_fan_out=True ) + local_name = output.get("step_template", {}).get("id", "step-0") + inherited = local.steps.get(local_name) child = node["children"][index] outcome, error = self.run( child, @@ -812,6 +814,10 @@ def run_item(index): if outcome in HALTING: halted.set() record = child["nodes"][0].get("result") + # Expose only results projected by the item traversal. Missing step + # implementations keep an internal retry record but publish nothing. + if local.steps.get(local_name) is inherited: + record = None if public and record is not None: # Projection only (E2): persisted by the next commit, rebuilt on replay. with self.state._lock: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 312addcac3..043d2612b3 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -1426,20 +1426,26 @@ def execute(self, config, context): assert state.step_results["missing"]["status"] == "completed" +@pytest.mark.parametrize("workers", [1, 2]) +@pytest.mark.parametrize("named", [False, True]) def test_unknown_fan_out_template_step_always_fails_despite_continue_on_error( - tmp_path, probe + tmp_path, monkeypatch, probe, workers, named ): + local_name = "missing" if named else "step-0" + alias_name = "missing" if named else "item" state = WorkflowEngine(tmp_path).execute( definition( "parent", [ + # An inherited same-name result must not become the item result. + {"id": local_name, "type": "probe", "value": "parent"}, { "id": "fan", "type": "fan-out", "items": [1, 2], - "max_concurrency": 2, + "max_concurrency": workers, "step": { - "id": "missing", + **({"id": "missing"} if named else {}), "type": "not-installed", "continue_on_error": True, }, @@ -1451,10 +1457,42 @@ def test_unknown_fan_out_template_step_always_fails_despite_continue_on_error( assert state.status == RunStatus.FAILED assert state.error == "Unknown step type: 'not-installed'" - assert not probe + assert probe == {local_name: 1} + assert set(state.step_results) == {local_name, "fan"} + saved = RunState.load(state.run_id, tmp_path) + assert saved.step_results == state.step_results + assert saved.step_results["fan"]["output"]["results"] == [{}] events = [entry["event"] for entry in state.log_entries] assert "step_failed" in events assert "step_continue_on_error" not in events + item_events = {} + for entry in state.log_entries: + if entry.get("step_id", "").startswith("fan:"): + item_events.setdefault(entry["step_id"], []).append(entry["event"]) + assert item_events + assert all( + events == ["step_started", "step_failed"] for events in item_events.values() + ) + + class Reinstalled(StepBase): + type_key = "not-installed" + + def execute(self, config, context): + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "not-installed", Reinstalled()) + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + assert probe == {local_name: 1, "after": 1} + assert state.step_results["fan"]["output"]["results"] == [ + {"value": 1}, {"value": 2} + ] + for index, value in enumerate([1, 2]): + assert state.step_results[f"fan:{alias_name}:{index}"]["output"] == { + "value": value + } + assert RunState.load(state.run_id, tmp_path).step_results == state.step_results def test_diamond_and_depth_limit(tmp_path, probe): From 6b776ce38c2411cb2521e04500bb63568b1c88e9 Mon Sep 17 00:00:00 2001 From: Markus Date: Mon, 28 Sep 2026 07:28:19 +0200 Subject: [PATCH 15/28] fix(workflows): preserve composed fan-out aliases Keep fan-out item aliases in the enclosing workflow context without exposing child workflow internals in root results. Normalize unnamed templates to the item ID for result aliases and lifecycle events, and recognize generated item aliases during fan-in validation. Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 20 ++-- src/specify_cli/workflows/engine.py | 15 ++- tests/workflows/test_composition_execution.py | 95 +++++++++++++++---- 3 files changed, 105 insertions(+), 25 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index e87e487b9a..8b20fb95fe 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -87,7 +87,8 @@ def child_steps(config, node, index): if kind == "workflow": return yaml.safe_load(node["binding"]["definition"])["steps"] if kind == "fan-out": - return [node["result"]["output"]["step_template"]] + template = node["result"]["output"]["step_template"] + return [{"id": "item", **template}] return steps_of(node["children"][0]) @@ -308,9 +309,8 @@ def project(self, context, name, result, *, public, qualified): self.state.step_results[qualified] = result def project_alias(self, context, name, result): - """Project a fan-out item result without exposing item internals.""" + """Project a fan-out item result into its enclosing workflow scope.""" context.steps[name] = result - self.state.step_results[name] = result def commit( self, @@ -793,13 +793,13 @@ def fan_out( workers = min(workers, len(items)) initial = deepcopy(context.steps) halted = threading.Event() - template_name = output.get("step_template", {}).get("id", "item") + template_name = child_steps(config, node, 0)[0]["id"] def run_item(index): local = replace( context, steps=deepcopy(initial), item=items[index], inside_fan_out=True ) - local_name = output.get("step_template", {}).get("id", "step-0") + local_name = template_name inherited = local.steps.get(local_name) child = node["children"][index] outcome, error = self.run( @@ -818,12 +818,18 @@ def run_item(index): # implementations keep an internal retry record but publish nothing. if local.steps.get(local_name) is inherited: record = None - if public and record is not None: + if record is not None: # Projection only (E2): persisted by the next commit, rebuilt on replay. with self.state._lock: self.project_alias( - context, f"{qualified}:{template_name}:{index}", record + context, + f"{qualified}:{template_name}:{index}", + record, ) + if public: + self.state.step_results[ + f"{qualified}:{template_name}:{index}" + ] = record return outcome, error, (record or {}).get("output", {}) results = [] diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index fff81a1b5e..f939f3c768 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -379,6 +379,7 @@ def _validate_steps( errors: list[str], input_defs: dict[str, Any] | None = None, inside_fan_out: bool = False, + fan_out_aliases: set[str] | None = None, ) -> None: """Recursively validate a list of steps. @@ -389,6 +390,8 @@ def _validate_steps( """ from . import STEP_REGISTRY + fan_out_aliases = set() if fan_out_aliases is None else fan_out_aliases + for step_config in steps: if not isinstance(step_config, dict): errors.append(f"Step must be a mapping, got {type(step_config).__name__}.") @@ -491,7 +494,11 @@ def _validate_steps( f"Fan-in step {step_id!r}: 'wait_for' references " f"itself; a fan-in cannot wait for its own results." ) - elif wid not in seen_ids: + elif wid not in seen_ids and not any( + wid.removeprefix(f"{alias}:").isdigit() + for alias in fan_out_aliases + if wid.startswith(f"{alias}:") + ): errors.append( f"Fan-in step {step_id!r}: 'wait_for' references " f"unknown or not-yet-declared step id {wid!r}." @@ -554,6 +561,7 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, + fan_out_aliases=fan_out_aliases, ) # Validate switch cases @@ -567,6 +575,7 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, + fan_out_aliases=fan_out_aliases, ) # Validate switch default @@ -578,6 +587,7 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, + fan_out_aliases=fan_out_aliases, ) # Validate fan-out nested step (template — not added to seen_ids @@ -593,6 +603,9 @@ def _validate_steps( inside_fan_out=True, ) errors.extend(fan_errors) + template_id = fan_step.get("id", "item") + if isinstance(template_id, str): + fan_out_aliases.add(f"{step_id}:{template_id}") # -- Run State Persistence ------------------------------------------------ diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 043d2612b3..a81221bb43 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -10,7 +10,12 @@ from specify_cli.workflows import STEP_REGISTRY from specify_cli.workflows.base import RunStatus, StepBase, StepResult, StepStatus -from specify_cli.workflows.engine import RunState, WorkflowDefinition, WorkflowEngine +from specify_cli.workflows.engine import ( + RunState, + WorkflowDefinition, + WorkflowEngine, + validate_workflow, +) def definition(name, steps, **fields): @@ -1016,17 +1021,7 @@ def commit(self, node=None, changes=None, **kwargs): assert committed == [] -def test_private_fan_out_context_has_no_qualified_aliases(tmp_path, monkeypatch, probe): - class Inspect(StepBase): - type_key = "inspect" - - def execute(self, config, context): - return StepResult( - StepStatus.COMPLETED, - output={"aliases": sorted(key for key in context.steps if ":" in key)}, - ) - - monkeypatch.setitem(STEP_REGISTRY, "inspect", Inspect()) +def test_private_fan_out_aliases_remain_available_to_child_fan_in(tmp_path, probe): install( tmp_path, definition( @@ -1035,19 +1030,55 @@ def execute(self, config, context): { "id": "fan", "type": "fan-out", - "items": [1], - "step": {"id": "template", "type": "probe"}, + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan:template:0", "fan:template:1"], }, - {"id": "inspect", "type": "inspect"}, ], - outputs={"aliases": {"value": "{{ steps.inspect.output.aliases }}"}}, + outputs={"results": {"value": "{{ steps.join.output.results }}"}}, ), ) state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) assert state.status == RunStatus.COMPLETED - assert state.step_results["call"]["output"]["aliases"] == [] + assert state.step_results["call"]["output"]["results"] == [ + {"value": 1}, + {"value": 2}, + ] + assert "fan:template:0" not in state.step_results + assert "fan:template:1" not in state.step_results + + +def test_fan_in_rejects_non_item_fan_out_alias(): + errors = validate_workflow( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe"}, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan:template:not-an-item"], + }, + ], + ) + ) + + assert any("unknown or not-yet-declared" in error for error in errors) def test_fan_out_in_later_loop_iteration_uses_qualified_aliases(tmp_path, probe): @@ -1119,6 +1150,36 @@ def test_fan_out_events_and_callbacks_use_qualified_item_ids(tmp_path, probe): ] +def test_unnamed_fan_out_templates_use_item_id_for_results_and_events(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"type": "probe", "value": "{{ item }}"}, + } + ], + ) + ) + + item_events = [ + entry["step_id"] + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + and entry["step_id"].startswith("fan:") + ] + assert callbacks == ["fan", "fan:item:0"] + assert item_events == ["fan:item:0", "fan:item:0"] + assert state.step_results["fan:item:0"]["output"] == {"value": 1} + + def test_loop_events_and_callbacks_use_qualified_iteration_ids(tmp_path, probe): callbacks = [] engine = WorkflowEngine(tmp_path) From 21d6eaf3203fe41bf48242c0bf2ef6bf675ae0fc Mon Sep 17 00:00:00 2001 From: Markus Wondrak Date: Mon, 28 Sep 2026 07:50:34 +0200 Subject: [PATCH 16/28] Refactor condition for raising ValueError on execution Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- src/specify_cli/workflows/_execution.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 8b20fb95fe..dd91d54260 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -209,10 +209,13 @@ def check_sequence(seq, steps=None, *, shared=False): check_sequence(child) if ( node["phase"] == "done" - and kind != "workflow" and any(nested["phase"] != "done" for nested in child["nodes"]) + and not ( + kind == "workflow" + and step.get("continue_on_error") is True + and result["status"] == "failed" + ) ): - raise ValueError("Completed execution has unfinished children") if node["phase"] == "outputs" and binding is None: raise ValueError("Output finalization requires a workflow binding") if node["phase"] == "children" and ( From 01bee4d3632ac702961ab33593e0132dc483061d Mon Sep 17 00:00:00 2001 From: Markus Date: Mon, 28 Sep 2026 08:21:33 +0200 Subject: [PATCH 17/28] fix(workflows): validate composition checkpoint boundaries Assisted-by: OpenCode (model: gpt-5.6-terra, autonomous) --- src/specify_cli/workflows/_execution.py | 19 ++++-- src/specify_cli/workflows/command_resume.py | 3 +- src/specify_cli/workflows/composition.py | 21 +++---- src/specify_cli/workflows/engine.py | 3 +- .../workflows/test_command_resume.py | 25 ++++++++ tests/test_workflows.py | 20 ++++++ tests/workflows/test_composition_execution.py | 61 ++++++++++++++++++- 7 files changed, 130 insertions(+), 22 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index dd91d54260..dd59aacf0d 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -184,7 +184,10 @@ def check_sequence(seq, steps=None, *, shared=False): if binding is not None: if kind != "workflow" or not isinstance(binding, dict): raise ValueError("Invalid workflow binding") - definition = yaml.safe_load(binding.get("definition", "")) + source = binding.get("definition") + if not isinstance(source, str): + raise ValueError("Invalid bound workflow definition or inputs") + definition = yaml.safe_load(source) if ( not isinstance(definition, dict) or not isinstance(definition.get("workflow"), dict) @@ -207,15 +210,19 @@ def check_sequence(seq, steps=None, *, shared=False): check_sequence(child, child_steps(step, node, index), shared=True) else: check_sequence(child) + handled_child_failure = ( + kind == "workflow" + and step.get("continue_on_error") is True + and node.get("outcome") == "completed" + and result is not None + and result["status"] == "failed" + ) if ( node["phase"] == "done" and any(nested["phase"] != "done" for nested in child["nodes"]) - and not ( - kind == "workflow" - and step.get("continue_on_error") is True - and result["status"] == "failed" - ) + and not handled_child_failure ): + raise ValueError("Completed execution has unfinished children") if node["phase"] == "outputs" and binding is None: raise ValueError("Output finalization requires a workflow binding") if node["phase"] == "children" and ( diff --git a/src/specify_cli/workflows/command_resume.py b/src/specify_cli/workflows/command_resume.py index 3be24f9582..585d531aec 100644 --- a/src/specify_cli/workflows/command_resume.py +++ b/src/specify_cli/workflows/command_resume.py @@ -20,6 +20,7 @@ def workflow_resume( ): """Resume a paused or failed workflow run.""" from . import load_custom_steps + from ._execution import CheckpointError from .engine import RunState, WorkflowEngine project_root = cli._require_specify_project() @@ -76,7 +77,7 @@ def workflow_resume( pre_state.installed_origin_tracked = True try: pre_state.save() - except OSError as exc: + except (OSError, CheckpointError) as exc: err.print(f"[red]Resume failed:[/red] {cli._escape_markup(str(exc))}") raise cli.typer.Exit(1) diff --git a/src/specify_cli/workflows/composition.py b/src/specify_cli/workflows/composition.py index de0de3a355..c7eff18028 100644 --- a/src/specify_cli/workflows/composition.py +++ b/src/specify_cli/workflows/composition.py @@ -51,16 +51,13 @@ def validate_call(config: dict[str, Any]) -> list[str]: ): errors.append("'workflow' must be an exact, safe, non-reserved workflow ID") mapping = config.get("input", {}) - if mapping is not None: - if not isinstance(mapping, dict) or any( - not isinstance(k, str) for k in mapping - ): - errors.append("'input' must be a mapping with string keys") - else: - try: - require_json(mapping) - except ValueError as exc: - errors.append(str(exc)) + if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): + errors.append("'input' must be a mapping with string keys") + else: + try: + require_json(mapping) + except ValueError as exc: + errors.append(str(exc)) return errors @@ -138,9 +135,7 @@ def bind_inputs( context: StepContext, previous: dict[str, Any] | None = None, ): - mapping = config.get("input") - if mapping is None: - mapping = {} + mapping = config.get("input", {}) if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): raise CallError("'input' must be a mapping with string keys") provided = { diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index f939f3c768..659f9c357f 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -495,7 +495,8 @@ def _validate_steps( f"itself; a fan-in cannot wait for its own results." ) elif wid not in seen_ids and not any( - wid.removeprefix(f"{alias}:").isdigit() + wid.removeprefix(f"{alias}:").isascii() + and wid.removeprefix(f"{alias}:").isdigit() for alias in fan_out_aliases if wid.startswith(f"{alias}:") ): diff --git a/tests/specify_cli/workflows/test_command_resume.py b/tests/specify_cli/workflows/test_command_resume.py index c4f0c050c0..21c12716d8 100644 --- a/tests/specify_cli/workflows/test_command_resume.py +++ b/tests/specify_cli/workflows/test_command_resume.py @@ -296,6 +296,31 @@ def test_resume_preload_io_error_is_reported_cleanly( assert "Resume failed" in result.output assert "permission [denied]" in result.output + def test_resume_legacy_migration_checkpoint_error_is_reported_cleanly( + self, project_dir, monkeypatch + ): + from typer.testing import CliRunner + from specify_cli import app + from specify_cli.workflows._execution import CheckpointError + from specify_cli.workflows.engine import RunState + + monkeypatch.chdir(project_dir) + runner = CliRunner() + run_id = self._install_and_run_gated(runner, app, project_dir) + state_path = project_dir / ".specify" / "workflows" / "runs" / run_id / "state.json" + data = json.loads(state_path.read_text(encoding="utf-8")) + data["installed_workflow_id"] = None + data.pop("installed_origin_tracked", None) + state_path.write_text(json.dumps(data), encoding="utf-8") + monkeypatch.setattr(RunState, "save", lambda _: (_ for _ in ()).throw(CheckpointError("disk full"))) + + result = runner.invoke(app, ["workflow", "resume", run_id]) + + assert result.exit_code != 0 + assert result.exception is None or isinstance(result.exception, SystemExit) + assert "Resume failed" in result.output + assert "disk full" in result.output + @pytest.mark.parametrize("malformation", ["non-object", "missing-run-id"]) def test_resume_preload_rejects_malformed_state_cleanly( self, project_dir, monkeypatch, malformation diff --git a/tests/test_workflows.py b/tests/test_workflows.py index 98b45f1e39..2155382f25 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -4789,6 +4789,26 @@ def test_non_string_wait_for_entry_is_rejected(self): "must be step-id strings" in e and "int" in e for e in errors ) + def test_non_ascii_fan_out_item_index_is_rejected(self): + errors = self._errors(""" +workflow: + id: wf + name: wf + version: "1.0.0" +steps: + - id: fan + type: fan-out + items: [one] + step: + id: item + type: command + command: speckit.implement + - id: collect + type: fan-in + wait_for: [fan:item:١] +""") + assert any("unknown or not-yet-declared step id 'fan:item:١'" in e for e in errors) + # ===== Workflow Definition Tests ===== diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index a81221bb43..d2caa7e358 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -279,7 +279,8 @@ def test_invalid_targets_are_handled_failures(tmp_path, probe, target): @pytest.mark.parametrize( - "mapping", [{"unknown": "x"}, {"value": []}, {"value": date(2026, 1, 1)}, [1]] + "mapping", + [{"unknown": "x"}, {"value": []}, {"value": date(2026, 1, 1)}, [1], None], ) def test_invalid_inputs_do_not_execute_child(tmp_path, probe, mapping): install( @@ -473,6 +474,9 @@ def test_legacy_resume_adapts_once(tmp_path, probe): lambda tree: tree["sequence"]["nodes"][0]["binding"].update( workflow_dir={} ), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + definition=42 + ), ], ) def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): @@ -559,6 +563,61 @@ def test_completed_container_with_unfinished_child_rejected(tmp_path, probe): assert probe["work"] == 1 +@pytest.mark.parametrize("child_phase", ["ready", "blocked"]) +def test_completed_workflow_call_with_unfinished_child_rejected( + tmp_path, probe, child_phase +): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}, {"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"}), {"id": "pause", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + call_node = data["execution"]["sequence"]["nodes"][0] + call_node.update(phase="done", outcome="completed") + if child_phase == "ready": + call_node["children"][0]["nodes"][1] = {"phase": "ready"} + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Completed execution has unfinished children"): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + + +def test_handled_failed_workflow_call_can_retain_blocked_child(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [ + {"id": "fail", "type": "probe", "status": "failed"}, + {"id": "unreached", "type": "probe"}, + ], + ), + ) + + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert RunState.load(state.run_id, tmp_path).status == RunStatus.COMPLETED + + def test_bound_call_without_source_path_resumes_with_null_workflow_dir( tmp_path, monkeypatch, probe ): From 9375b86e8e09406380b1e60f69926622511e487a Mon Sep 17 00:00:00 2001 From: Markus Date: Mon, 28 Sep 2026 19:00:48 +0200 Subject: [PATCH 18/28] fix(workflows): make fan-out item aliases reporting-only Fan-out item aliases (fan:template:N) are not valid fan-in wait_for targets. In a loop iteration >= 1 the authored alias is never refreshed, so a fan-in that waited on it silently joined the previous iteration's values. Joins now use the fan-out step ID and its ordered output.results instead. Revert the fan_out_aliases validation allowance so wait_for again requires declared step IDs (matching main), and reject ':' wait_for entries at runtime in FanInStep.execute() so unvalidated runs fail loudly rather than joining a stale or empty item result. Assisted-by: OpenCode (model: deepseek-v4.1-flash, autonomous) --- design/workflow-step.md | 5 +- docs/reference/workflows.md | 5 +- src/specify_cli/workflows/engine.py | 16 +-- .../workflows/step/fan_in/__init__.py | 19 +++ tests/workflows/test_composition_execution.py | 114 ++++++++++++++---- workflows/README.md | 5 + 6 files changed, 126 insertions(+), 38 deletions(-) diff --git a/design/workflow-step.md b/design/workflow-step.md index c03b0b1ec9..9011b3de8d 100644 --- a/design/workflow-step.md +++ b/design/workflow-step.md @@ -53,7 +53,10 @@ through their saved top-level index. Fan-out items have independent expression contexts. Their internal results do not enter the shared parent context; the parent receives the qualified item -result and the fan-out step's ordered `output.results`. A side effect performed +result and the fan-out step's ordered `output.results`. Fan-out item aliases are +reporting-only; a `fan-in` `wait_for` targets declared step IDs (the fan-out +step's own `id`) and reads its ordered `output.results`, never a generated item +alias. A side effect performed before its completion checkpoint can still repeat after an interruption, so the guarantee is at-least-once rather than exactly-once. diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index cb540a7830..1f3f74fa2d 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -667,7 +667,10 @@ step ID is a local expression alias, not a global execution ID. Fan-out items have independent alias contexts and ordered item results. Public reporting uses qualified occurrence IDs where needed, for example `fan:template:0` for a fan-out item and `loop:step:1` for a later loop iteration. Qualified IDs are -not expression names. +not expression names. Fan-out item aliases are reporting-only: a `fan-in` +`wait_for` references declared step IDs, in particular the fan-out step's own +`id`, whose ordered item results are available as +`steps..output.results`. New runs persist selected branches, dynamic custom-step expansions, loop iterations, and fan-out items. Resume retains completed work without diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 659f9c357f..fff81a1b5e 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -379,7 +379,6 @@ def _validate_steps( errors: list[str], input_defs: dict[str, Any] | None = None, inside_fan_out: bool = False, - fan_out_aliases: set[str] | None = None, ) -> None: """Recursively validate a list of steps. @@ -390,8 +389,6 @@ def _validate_steps( """ from . import STEP_REGISTRY - fan_out_aliases = set() if fan_out_aliases is None else fan_out_aliases - for step_config in steps: if not isinstance(step_config, dict): errors.append(f"Step must be a mapping, got {type(step_config).__name__}.") @@ -494,12 +491,7 @@ def _validate_steps( f"Fan-in step {step_id!r}: 'wait_for' references " f"itself; a fan-in cannot wait for its own results." ) - elif wid not in seen_ids and not any( - wid.removeprefix(f"{alias}:").isascii() - and wid.removeprefix(f"{alias}:").isdigit() - for alias in fan_out_aliases - if wid.startswith(f"{alias}:") - ): + elif wid not in seen_ids: errors.append( f"Fan-in step {step_id!r}: 'wait_for' references " f"unknown or not-yet-declared step id {wid!r}." @@ -562,7 +554,6 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, - fan_out_aliases=fan_out_aliases, ) # Validate switch cases @@ -576,7 +567,6 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, - fan_out_aliases=fan_out_aliases, ) # Validate switch default @@ -588,7 +578,6 @@ def _validate_steps( errors, input_defs, inside_fan_out=inside_fan_out, - fan_out_aliases=fan_out_aliases, ) # Validate fan-out nested step (template — not added to seen_ids @@ -604,9 +593,6 @@ def _validate_steps( inside_fan_out=True, ) errors.extend(fan_errors) - template_id = fan_step.get("id", "item") - if isinstance(template_id, str): - fan_out_aliases.add(f"{step_id}:{template_id}") # -- Run State Persistence ------------------------------------------------ diff --git a/src/specify_cli/workflows/step/fan_in/__init__.py b/src/specify_cli/workflows/step/fan_in/__init__.py index ddcc2afcbb..ca5967ada4 100644 --- a/src/specify_cli/workflows/step/fan_in/__init__.py +++ b/src/specify_cli/workflows/step/fan_in/__init__.py @@ -86,6 +86,25 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: output={"results": []}, ) + # Generated fan-out item aliases (``fan:template:0``) are reporting-only: + # declared step IDs cannot contain ':' (engine validation rejects them), + # so a ':' entry can never match a real step. The engine does not + # auto-validate before ``execute``, so reject it loudly here too rather + # than silently joining a stale or empty item result. + aliased_entries = [w for w in wait_for if ":" in w] + if aliased_entries: + first = aliased_entries[0] + return StepResult( + status=StepStatus.FAILED, + error=( + f"Fan-in step {config.get('id', '?')!r}: 'wait_for' entry " + f"{first!r} looks like a fan-out item alias; wait on the " + f"fan-out step id instead — its 'output.results' holds the " + f"ordered item results." + ), + output={"results": []}, + ) + # Collect results from referenced steps results = [] for step_id in wait_for: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index d2caa7e358..e115ef4649 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -778,7 +778,7 @@ def test_expansion_resume_preserves_completed_work(tmp_path, probe, kind): assert probe["prepare"] == (1 if kind == "if" else 2) -def test_replay_restores_fan_out_aliases_from_completed_if(tmp_path, probe): +def test_replay_restores_fan_out_item_aliases_from_completed_if(tmp_path, probe): state = WorkflowEngine(tmp_path).execute( definition( "parent", @@ -801,7 +801,12 @@ def test_replay_restores_fan_out_aliases_from_completed_if(tmp_path, probe): ], }, {"id": "wait", "type": "probe", "await": True}, - {"id": "join", "type": "fan-in", "wait_for": ["fan:template:0"]}, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, ], inputs={"approve": {"type": "boolean", "default": False}}, ) @@ -811,7 +816,8 @@ def test_replay_restores_fan_out_aliases_from_completed_if(tmp_path, probe): state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) assert state.status == RunStatus.COMPLETED - assert state.step_results["join"]["output"]["results"] == [{"value": 1}] + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["join"]["output"]["merged"] == [{"value": 1}] def test_resume_restores_completed_fan_out_item_aliases(tmp_path, monkeypatch, probe): @@ -837,7 +843,8 @@ def execute(self, config, context): { "id": "join", "type": "fan-in", - "wait_for": ["fan:template:0", "fan:template:1"], + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, }, ], inputs={"approve": {"type": "boolean", "default": False}}, @@ -848,7 +855,9 @@ def execute(self, config, context): state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) assert state.status == RunStatus.COMPLETED - assert state.step_results["join"]["output"]["results"] == [ + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["fan:template:1"]["output"] == {"value": 2} + assert state.step_results["join"]["output"]["merged"] == [ {"value": 1}, {"value": 2}, ] @@ -1080,7 +1089,7 @@ def commit(self, node=None, changes=None, **kwargs): assert committed == [] -def test_private_fan_out_aliases_remain_available_to_child_fan_in(tmp_path, probe): +def test_composed_fan_out_joins_container_results(tmp_path, probe): install( tmp_path, definition( @@ -1099,10 +1108,11 @@ def test_private_fan_out_aliases_remain_available_to_child_fan_in(tmp_path, prob { "id": "join", "type": "fan-in", - "wait_for": ["fan:template:0", "fan:template:1"], + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, }, ], - outputs={"results": {"value": "{{ steps.join.output.results }}"}}, + outputs={"results": {"value": "{{ steps.join.output.merged }}"}}, ), ) @@ -1117,27 +1127,89 @@ def test_private_fan_out_aliases_remain_available_to_child_fan_in(tmp_path, prob assert "fan:template:1" not in state.step_results -def test_fan_in_rejects_non_item_fan_out_alias(): - errors = validate_workflow( +def test_fan_in_rejects_fan_out_item_alias(): + for alias in ("fan:template:0", "fan:template:not-an-item"): + errors = validate_workflow( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe"}, + }, + {"id": "join", "type": "fan-in", "wait_for": [alias]}, + ], + ) + ) + + assert any("unknown or not-yet-declared" in error for error in errors), alias + + +def test_fan_in_rejects_item_alias_at_runtime(): + from specify_cli.workflows.base import StepContext + from specify_cli.workflows.step.fan_in import FanInStep + + result = FanInStep().execute( + {"id": "join", "type": "fan-in", "wait_for": ["fan:template:0"]}, + StepContext(), + ) + + assert result.status == StepStatus.FAILED + assert "fan-out item alias" in (result.error or "") + + +def test_fan_in_container_join_in_loop_sees_current_iteration(tmp_path, monkeypatch, probe): + calls = [] + + class Varying(StepBase): + type_key = "varying" + + def execute(self, config, context): + calls.append(1) + items = [10, 20] if len(calls) == 1 else [30, 40] + return StepResult(output={"items": items}) + + monkeypatch.setitem(STEP_REGISTRY, "varying", Varying()) + state = WorkflowEngine(tmp_path).execute( definition( "parent", [ { - "id": "fan", - "type": "fan-out", - "items": [1], - "step": {"id": "template", "type": "probe"}, - }, - { - "id": "join", - "type": "fan-in", - "wait_for": ["fan:template:not-an-item"], - }, + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [ + {"id": "varying", "type": "varying"}, + { + "id": "fan", + "type": "fan-out", + "items": "{{ steps.varying.output.items }}", + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, + ], + } ], ) ) - assert any("unknown or not-yet-declared" in error for error in errors) + assert state.status == RunStatus.COMPLETED + assert state.step_results["join"]["output"]["merged"] == [ + {"value": 30}, + {"value": 40}, + ] def test_fan_out_in_later_loop_iteration_uses_qualified_aliases(tmp_path, probe): diff --git a/workflows/README.md b/workflows/README.md index b343c9ac4a..4ece1d330c 100644 --- a/workflows/README.md +++ b/workflows/README.md @@ -320,6 +320,11 @@ Aggregate results from fan-out steps: output: {} ``` +`wait_for` references declared step IDs. For a fan-out, use the fan-out step's +own `id` (`parallel-impl` above); its ordered item results are available as +`steps..output.results`. Generated item aliases such as +`fan:template:0` are reporting-only and are not valid `wait_for` targets. + ## Error Handling By default, any step that returns `StepResult(status=StepStatus.FAILED, ...)` From 3c503260f8eca9b8822d3b38f671c26320b75e33 Mon Sep 17 00:00:00 2001 From: Markus Wondrak Date: Tue, 29 Sep 2026 09:47:24 +0200 Subject: [PATCH 19/28] fix(workflows): stop logging after checkpoint failure; treat unknown child steps as call failures RunState.append_log() now checks _checkpoint_failed under the state lock (lock order _lock -> _log_lock), so no concurrent fan-out worker can append a completion or nested start event after another worker's checkpoint failed. This restores the documented guarantee that a checkpoint failure prevents further writes. Remove the error-text classification of unavailable child implementations at the workflow-call boundary. A missing implementation stays terminal where the step occurs (root and fan-out, as on main); at a workflow call it is a reported child failure, whether detected at bind time or after binding, so call-level continue_on_error treats both consistently and step-controlled error strings are no longer interpreted. Assisted-by: GitHub Copilot (model: claude-opus-5.5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/reference/workflows.md | 6 +- src/specify_cli/workflows/_execution.py | 14 +- src/specify_cli/workflows/engine.py | 21 ++- tests/workflows/test_composition_execution.py | 162 +++++++++++++++++- 4 files changed, 175 insertions(+), 28 deletions(-) diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 1f3f74fa2d..5fd84db01a 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -593,8 +593,10 @@ and call-boundary contract failures, such as an unavailable target, invalid mapped input, cycle, depth limit, or invalid declared output. It does not catch step exceptions, expression errors (including `from_json` errors in `input:` or `outputs:`), interruptions, or checkpoint failures; those propagate exactly as -they do at the root. An unavailable step implementation is terminal at every -boundary. Pauses and explicit aborts always stop execution. +they do at the root. An unavailable step implementation is terminal where the +step occurs; at a workflow call it is a reported child failure, whether it is +detected while binding the target or later. Pauses and explicit aborts always +stop execution. Nested composition is allowed, but repeated workflow IDs on the active call path are cycles. Diamonds are allowed. The maximum included depth is 16, with diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index dd59aacf0d..d75c4e4148 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -26,16 +26,8 @@ HALTING = {"paused", "failed", "aborted"} -UNKNOWN_STEP_PREFIX = "Unknown step type: " - - def unknown_step_error(kind): - return f"{UNKNOWN_STEP_PREFIX}{kind!r}" - - -def is_unknown_step_error(error): - """A missing implementation stays terminal across workflow boundaries.""" - return isinstance(error, str) and error.startswith(UNKNOWN_STEP_PREFIX) + return f"Unknown step type: {kind!r}" class CheckpointError(RuntimeError): @@ -774,10 +766,6 @@ def workflow( else StepStatus.FAILED ) result = StepResult(status, output=output, error=error) - # An unavailable child implementation is terminal at every workflow - # boundary, matching direct execution of the same step. - if is_unknown_step_error(result.error): - config = {**config, "continue_on_error": False} return self.finish( config, node, diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index fff81a1b5e..b36df3921a 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -712,9 +712,9 @@ def __init__( # Reentrant so an execution transition can update views and call save() # under the same lock. Step implementations run outside that lock. # Serializes append_log's list append + log.jsonl write so concurrent - # fan-out workers cannot interleave or corrupt log lines. Kept separate - # from _lock so frequent logging never contends with state saves; since - # append_log is never called while _lock is held, the two never nest. + # fan-out workers cannot interleave or corrupt log lines. append_log + # takes _lock first to observe a checkpoint failure atomically, so the + # order is always _lock -> _log_lock and the two cannot deadlock. self._log_lock = threading.Lock() self.inputs: dict[str, Any] = {} self.workflow_dir: str | None = None @@ -928,14 +928,21 @@ def append_log(self, entry: dict[str, Any]) -> None: Held under ``_log_lock`` so concurrent fan-out workers serialize their list append and ``log.jsonl`` write rather than interleaving lines. + The checkpoint-failure check shares ``_lock`` with ``save()``, so no + worker can log once any checkpoint of this run has failed. """ entry["timestamp"] = datetime.now(timezone.utc).isoformat() runs_dir = self.runs_dir runs_dir.mkdir(parents=True, exist_ok=True) - with self._log_lock: - self.log_entries.append(entry) - with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f: - f.write(json.dumps(entry) + "\n") + with self._lock: + if self._checkpoint_failed: + from ._execution import CheckpointError + + raise CheckpointError("A previous checkpoint failed; reload the run") + with self._log_lock: + self.log_entries.append(entry) + with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f: + f.write(json.dumps(entry) + "\n") # -- Workflow Engine ------------------------------------------------------ diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index e115ef4649..578cd1c0d9 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -681,6 +681,79 @@ def write(path, data): assert probe["work"] == 1 +def test_checkpoint_failure_stops_concurrent_fan_out_logging(tmp_path, monkeypatch): + import specify_cli.workflows._execution as execution + from specify_cli.workflows._execution import CheckpointError + + items = {} + first_committed = threading.Event() + second_failed = threading.Event() + + class Sync(StepBase): + type_key = "sync" + + def execute(self, config, context): + items[threading.current_thread()] = context.item + if context.item == 2: + assert first_committed.wait(5) + return StepResult(output={"value": context.item}) + + original_commit = execution.Execution.commit + original_write = RunState._atomic_write_json + + def commit(self, *args, **kwargs): + item = items.get(threading.current_thread()) + try: + original_commit(self, *args, **kwargs) + except CheckpointError: + if item == 2: + second_failed.set() + raise + if item == 1 and not first_committed.is_set(): + # Item 1 committed and released the lock; item 2 now fails its + # checkpoint before item 1 emits its completion event. + first_committed.set() + assert second_failed.wait(5) + + def write(path, data): + if items.get(threading.current_thread()) == 2: + raise OSError("disk full") + original_write(path, data) + + monkeypatch.setitem(STEP_REGISTRY, "sync", Sync()) + monkeypatch.setattr(execution.Execution, "commit", commit) + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + with pytest.raises(CheckpointError): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"type": "sync"}, + } + ], + ), + run_id="concurrent-fault", + ) + + assert second_failed.is_set() + log = tmp_path / ".specify/workflows/runs/concurrent-fault/log.jsonl" + events = [json.loads(line) for line in log.read_text().splitlines()] + item_events = [ + (entry["event"], entry["step_id"]) + for entry in events + if entry.get("step_id", "").startswith("fan:") + ] + assert sorted(item_events) == [ + ("step_started", "fan:item:0"), + ("step_started", "fan:item:1"), + ] + + def test_completion_log_failure_does_not_replay_committed_step( tmp_path, monkeypatch, probe ): @@ -1540,7 +1613,7 @@ def resolve(self, workflow_id): @pytest.mark.parametrize( "child_continue, call_continue", [(False, False), (True, False), (False, True)] ) -def test_unknown_child_step_always_fails_despite_continue_on_error( +def test_unknown_child_step_is_a_reported_call_failure( tmp_path, monkeypatch, probe, child_continue, call_continue ): class TemporarilyInstalled(StepBase): @@ -1582,17 +1655,94 @@ def execute(self, config, context): monkeypatch.delitem(STEP_REGISTRY, "temporarily-installed") state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) - assert state.status == RunStatus.FAILED - assert state.error == "Unknown step type: 'temporarily-installed'" - assert probe == {"wait": 2} + error = "Unknown step type: 'temporarily-installed'" events = [entry["event"] for entry in state.log_entries] - assert "step_failed" in events - assert "step_continue_on_error" not in events + # The missing step itself is terminal inside the child, as at the root. assert [ entry["event"] for entry in state.log_entries if entry.get("step_id") == "missing" and entry.get("workflow_id") == "child" ] == ["step_started", "step_failed"] + if call_continue: + # At the call boundary it is a reported child failure. + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert state.step_results["call"]["output"]["error"] == error + assert probe == {"wait": 2, "after": 1} + assert "step_continue_on_error" in events + else: + assert state.status == RunStatus.FAILED + assert state.error == error + assert probe == {"wait": 2} + assert "step_continue_on_error" not in events + + +@pytest.mark.parametrize("call_continue", [False, True]) +def test_unknown_child_step_at_bind_is_a_reported_call_failure( + tmp_path, monkeypatch, probe, call_continue +): + install( + tmp_path, + definition("child", [{"id": "missing", "type": "later-installed"}]), + ) + parent = definition( + "parent", + [call(continue_on_error=call_continue), {"id": "after", "type": "probe"}], + ) + state = WorkflowEngine(tmp_path).execute(parent, run_id="bind") + + assert "invalid type 'later-installed'" in state.step_results["call"]["error"] + assert "binding" not in state.execution["sequence"]["nodes"][0] + if call_continue: + # Same treatment as an implementation that disappears after binding. + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert probe == {"after": 1} + return + + assert state.status == RunStatus.FAILED + assert not probe + + class LaterInstalled(StepBase): + type_key = "later-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "later-installed", LaterInstalled()) + state = WorkflowEngine(tmp_path).resume("bind") + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "completed" + assert probe == {"after": 1} + + +def test_step_reported_unknown_type_error_is_handled_like_any_failure( + tmp_path, monkeypatch, probe +): + class Mimic(StepBase): + type_key = "mimic" + + def execute(self, config, context): + return StepResult(StepStatus.FAILED, error="Unknown step type: 'mimic'") + + monkeypatch.setitem(STEP_REGISTRY, "mimic", Mimic()) + install(tmp_path, definition("child", [{"id": "work", "type": "mimic"}])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "direct", "type": "mimic", "continue_on_error": True}, + call(continue_on_error=True), + {"id": "after", "type": "probe"}, + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["direct"]["status"] == "failed" + assert state.step_results["call"]["status"] == "failed" + assert probe == {"after": 1} def test_unknown_step_type_resumes_after_reinstall(tmp_path, monkeypatch, probe): From f2dca00fed62c9f5fea6512cd99b90327945593c Mon Sep 17 00:00:00 2001 From: Markus Wondrak Date: Tue, 29 Sep 2026 14:51:22 +0200 Subject: [PATCH 20/28] test(workflows): cover resume snapshot check with native YAML scalars Add a resume regression for a workflow snapshot containing `.nan` (max_concurrency sequential fallback) and reordered keys, and cover the previously untested rejection of a changed workflow snapshot. Both pass on the existing comparison: PyYAML decodes `.nan` to a shared object, so the snapshots compare equal without a production change. Assisted-by: GitHub Copilot (model: claude-opus-5.5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/workflows/test_composition_execution.py | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 578cd1c0d9..57f748b99b 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -2482,3 +2482,61 @@ def execute(self, config, context): state = WorkflowEngine(tmp_path).resume(state.run_id) assert state.status == RunStatus.ABORTED assert counts == {0: 2, 1: 1} + + +NAN_SNAPSHOT = """ +schema_version: "1.0" +workflow: + id: nan-snapshot + name: NaN Snapshot + version: "1.0.0" +inputs: + approve: + type: boolean + default: false +steps: + - id: fan + type: fan-out + items: [1, 2] + max_concurrency: .nan + step: + type: probe + value: "{{ item }}" + - id: wait + type: probe + await: true +""" + + +def test_resume_accepts_native_yaml_nan_in_snapshot(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition.from_string(NAN_SNAPSHOT), run_id="nan" + ) + assert state.status == RunStatus.PAUSED + copy = tmp_path / ".specify/workflows/runs/nan/workflow.yml" + data = yaml.safe_load(copy.read_text(encoding="utf-8")) + # Key order is not part of the snapshot identity. + data["steps"][0] = dict(reversed(list(data["steps"][0].items()))) + copy.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") + + state = WorkflowEngine(tmp_path).resume("nan", {"approve": True}) + + assert state.status == RunStatus.COMPLETED, state.error + assert state.step_results["fan"]["output"]["results"] == [ + {"value": 1}, {"value": 2} + ] + assert probe == {"item": 2, "wait": 2} + + +def test_resume_rejects_changed_workflow_snapshot(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition.from_string(NAN_SNAPSHOT), run_id="changed" + ) + assert state.status == RunStatus.PAUSED + copy = tmp_path / ".specify/workflows/runs/changed/workflow.yml" + data = yaml.safe_load(copy.read_text(encoding="utf-8")) + data["steps"][0]["items"] = [1, 2, 3] + copy.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") + + with pytest.raises(ValueError, match="root sequence differs"): + WorkflowEngine(tmp_path).resume("changed", {"approve": True}) \ No newline at end of file From 97874f3c51d785e3f0898e4b8363c3db3cdc5328 Mon Sep 17 00:00:00 2001 From: Markus Wondrak Date: Tue, 29 Sep 2026 15:40:30 +0200 Subject: [PATCH 21/28] Refactor executor run with error handling Refactor execution handling to ensure active step is updated even if an error occurs during execution. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- src/specify_cli/workflows/engine.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index b36df3921a..d572c2c50e 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -1249,10 +1249,14 @@ def _execute_steps( context.steps = deepcopy(state.execution.get("initial", {})) tree = state.execution["sequence"] executor = Execution(self, state, registry, rebind=rebind) - outcome, error = executor.run(tree, context, (state.workflow_id,), root=True) - active = active_step(state.execution) - if active is not None: - state.current_step_id = active[0][-1] + try: + outcome, error = executor.run( + tree, context, (state.workflow_id,), root=True + ) + finally: + active = active_step(state.execution) + if active is not None: + state.current_step_id = active[0][-1] state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) state.error = error From 2c0e141504cd3ac1889cf548360b6805eda1a0cd Mon Sep 17 00:00:00 2001 From: Markus Date: Tue, 29 Sep 2026 19:45:19 +0200 Subject: [PATCH 22/28] fix(workflows): report qualified current_step_id; keep typed gate messages; halt fan-out on worker exceptions current_step_id again uses the occurrence ID that events and step_results use, as on main: a paused or failed root fan-out item reports `fan:item:0` and a later loop iteration `loop:body:1`. Inside a workflow call, the ID is relative to the called workflow. walk_execution() now derives this ID with the executor's qualification rules, so the post-run reconciliation (kept for exception paths) and the gate payload's step_id report the same value. Gate `message` keeps its typed template result (for example 42 stays an int). Only values the JSON checkpoint cannot persist, such as native YAML dates, are stored as text; presentation still stringifies. A fan-out worker exception now sets the halt flag before re-raising, so earlier items finishing in order no longer dispatch further items after a later item has already raised. Assisted-by: GitHub Copilot CLI (model: claude-opus-5.5, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/reference/workflows.md | 7 +- src/specify_cli/workflows/_commands.py | 3 +- src/specify_cli/workflows/_execution.py | 53 +++-- src/specify_cli/workflows/engine.py | 2 +- .../workflows/step/gate/__init__.py | 9 +- tests/workflows/test_composition_execution.py | 202 +++++++++++++++++- 6 files changed, 255 insertions(+), 21 deletions(-) diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 5fd84db01a..2e15b5e814 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -694,8 +694,11 @@ cause ordinary step failures. Legacy runs without a tree enter through their saved root index, then use tree-backed resume. Resume is available only for runs in the `paused` or `failed` state. -`current_step_index` is the root-sequence index; `current_step_id` is the local -ID of the active leaf. Structured run/status output includes `workflow_scopes` +`current_step_index` is the root-sequence index; `current_step_id` is the +occurrence ID of the active leaf, matching the event `step_id` (for example +`fan:item:0` for a fan-out item or `loop:body:1` for a later loop iteration). +Inside a workflow call, IDs are relative to the called workflow. Structured +run/status output includes `workflow_scopes` summaries when calls exist and reports an active nested gate with its `scope_path`. Private log events add `workflow_id` and `execution_path` to the qualified `step_id`. A step emits completion after its checkpoint; containers diff --git a/src/specify_cli/workflows/_commands.py b/src/specify_cli/workflows/_commands.py index e12d8d06a3..e977d4552b 100644 --- a/src/specify_cli/workflows/_commands.py +++ b/src/specify_cli/workflows/_commands.py @@ -992,8 +992,7 @@ def _gate_outcome(state: Any) -> dict[str, Any] | None: active = active_step(state.execution) if active is None: return None - path, node = active - step_id = path[-1] + path, node, step_id = active scope_path = path[:-1] step = node.get("result") if not isinstance(step, dict) or not _is_gate_step(step): diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index d75c4e4148..86e704cbd6 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -94,16 +94,29 @@ def child_path(config, path, item): def walk_execution( - seq, path=(), *, children_first=False, skip_done=False, steps=None + seq, + path=(), + *, + children_first=False, + skip_done=False, + steps=None, + loop_alias=None, ): - """Yield execution occurrences with one shared path rule.""" + """Yield execution occurrences with one shared path and ID rule. + + Each item is ``(config, node, path, step_id)``; ``step_id`` is the + occurrence ID used by events, relative to the enclosing workflow scope. + """ steps = steps_of(seq) if steps is None else steps for index, (config, node) in enumerate(zip(steps, seq["nodes"])): if skip_done and node["phase"] == "done": continue - here = [*path, config.get("id", f"step-{index}")] + name = config.get("id", f"step-{index}") + here = [*path, name] + qualified = qualified_id(name, loop_alias) + kind = config.get("type", "command") if not children_first: - yield config, node, here + yield config, node, here, qualified for item, child in enumerate(node.get("children", [])): yield from walk_execution( child, @@ -111,9 +124,14 @@ def walk_execution( children_first=children_first, skip_done=skip_done, steps=child_steps(config, node, item), + loop_alias=( + (qualified, item) + if kind == "fan-out" + else loop_alias_for(kind, qualified, item) + ), ) if children_first: - yield config, node, here + yield config, node, here, qualified def validate_execution(tree: Any) -> None: @@ -244,18 +262,21 @@ def check_sequence(seq, steps=None, *, shared=False): def active_step(tree): - """First unfinished leaf in execution order, including nested workflow scopes.""" - for _, node, path in walk_execution( + """First unfinished leaf in execution order, including nested workflow scopes. + + Returns ``(path, node, step_id)``. + """ + for _, node, path, step_id in walk_execution( tree["sequence"], children_first=True, skip_done=True ): - return path, node + return path, node, step_id return None def scope_summaries(tree): """Report workflow boundaries without exposing private inputs or results.""" summaries = [] - for _, node, path in walk_execution(tree["sequence"]): + for _, node, path, _ in walk_execution(tree["sequence"]): binding = node.get("binding") if binding: output = node.get("result", {}).get("output", {}) @@ -452,7 +473,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): if node["phase"] in {"ready", "blocked"}: with self.state._lock: - self.state.current_step_id = name + self.state.current_step_id = qualified self.emit( "step_started", qualified, @@ -838,8 +859,16 @@ def run_item(index): if outcome in HALTING: return outcome, error, results return "completed", None, results + + def run_item_guarded(index): + try: + return run_item(index) + except BaseException: + halted.set() + raise + with ThreadPoolExecutor(max_workers=workers) as pool: - futures = {i: pool.submit(run_item, i) for i in range(workers)} + futures = {i: pool.submit(run_item_guarded, i) for i in range(workers)} for index in range(len(items)): try: outcome, error, value = futures.pop(index).result() @@ -854,5 +883,5 @@ def run_item(index): return outcome, error, results following = index + workers if following < len(items) and not halted.is_set(): - futures[following] = pool.submit(run_item, following) + futures[following] = pool.submit(run_item_guarded, following) return "completed", None, results diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index d572c2c50e..9224a1d6f9 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -1256,7 +1256,7 @@ def _execute_steps( finally: active = active_step(state.execution) if active is not None: - state.current_step_id = active[0][-1] + state.current_step_id = active[2] state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) state.error = error diff --git a/src/specify_cli/workflows/step/gate/__init__.py b/src/specify_cli/workflows/step/gate/__init__.py index 4297016d36..86dec595b2 100644 --- a/src/specify_cli/workflows/step/gate/__init__.py +++ b/src/specify_cli/workflows/step/gate/__init__.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import re import sys from pathlib import Path @@ -40,9 +41,11 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: message = config.get("message", "Review required.") if isinstance(message, str) and "{{" in message: message = evaluate_expression(message, context) - # Step results are persisted as JSON. Keep the display value serializable - # while _compose_prompt() independently accepts synthetic non-string data. - if message is not None: + # Keep JSON values typed; only values the JSON checkpoint cannot + # persist (e.g. native YAML dates) are stored as text. + try: + json.dumps(message) + except (TypeError, ValueError): message = str(message) options = config.get("options", ["approve", "reject"]) diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 57f748b99b..20f7f17bef 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -1909,6 +1909,8 @@ def test_nested_gate_reporting_uses_active_occurrence(tmp_path, probe): ) assert state.status == RunStatus.PAUSED payload = _workflow_run_payload(RunState.load(state.run_id, tmp_path)) + # Private scopes report the child-relative occurrence ID (own namespace). + assert payload["current_step_id"] == "review" assert payload["gate"]["step_id"] == "review" assert payload["gate"]["scope_path"] == ["route", "call"] assert payload["workflow_scopes"][0]["status"] == "paused" @@ -2539,4 +2541,202 @@ def test_resume_rejects_changed_workflow_snapshot(tmp_path, probe): copy.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") with pytest.raises(ValueError, match="root sequence differs"): - WorkflowEngine(tmp_path).resume("changed", {"approve": True}) \ No newline at end of file + WorkflowEngine(tmp_path).resume("changed", {"approve": True}) + +def test_paused_fan_out_item_reports_qualified_current_step_id(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "item", "type": "probe", "status": "paused"}, + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + assert state.current_step_id == "fan:item:0" + assert "fan:item:0" in state.step_results + assert RunState.load(state.run_id, tmp_path).current_step_id == "fan:item:0" + assert callbacks[-1] == "fan:item:0" + + +def test_paused_later_loop_iteration_reports_qualified_current_step_id( + tmp_path, monkeypatch +): + calls = Counter() + + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + calls[config["id"]] += 1 + if calls[config["id"]] == 2: + return StepResult(StepStatus.PAUSED) + return StepResult(output={}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 3, + "steps": [{"id": "body", "type": "pause-second"}], + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + assert state.current_step_id == "loop:body:1" + assert "loop:body:1" in state.step_results + assert RunState.load(state.run_id, tmp_path).current_step_id == "loop:body:1" + + +def test_failed_fan_out_item_exception_reports_qualified_current_step_id( + tmp_path, monkeypatch +): + class Boom(StepBase): + type_key = "boom" + + def execute(self, config, context): + raise RuntimeError("boom") + + monkeypatch.setitem(STEP_REGISTRY, "boom", Boom()) + engine = WorkflowEngine(tmp_path) + with pytest.raises(RuntimeError, match="boom"): + engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "blast", "type": "boom"}, + } + ], + ), + run_id="boom", + ) + + loaded = RunState.load("boom", tmp_path) + assert loaded.status == RunStatus.FAILED + assert loaded.current_step_id == "fan:blast:0" + + +def test_paused_fan_out_gate_payload_reports_qualified_step_id(tmp_path): + from specify_cli.workflows._commands import _workflow_run_payload + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "review", "type": "gate", "message": "Approve"}, + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + payload = _workflow_run_payload(RunState.load(state.run_id, tmp_path)) + assert payload["current_step_id"] == "fan:review:0" + assert payload["gate"]["step_id"] == "fan:review:0" + assert payload["gate"]["message"] == "Approve" + + +def test_gate_message_keeps_typed_template_result(tmp_path): + from specify_cli.workflows._commands import _workflow_run_payload + + workflow = definition( + "parent", + [{"id": "gate", "type": "gate", "message": "{{ inputs.notice }}"}], + inputs={"notice": {"type": "number"}}, + ) + assert validate_workflow(workflow) == [] + + state = WorkflowEngine(tmp_path).execute(workflow, {"notice": 42}) + + assert state.status == RunStatus.PAUSED + loaded = RunState.load(state.run_id, tmp_path) + message = loaded.step_results["gate"]["output"]["message"] + assert message == 42 + assert isinstance(message, int) + assert _workflow_run_payload(loaded)["gate"]["message"] == "42" + + +def test_gate_message_stores_non_json_literal_as_text(tmp_path): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "gate", "type": "gate", "message": date(2026, 1, 1)}]) + ) + + assert state.status == RunStatus.PAUSED + loaded = RunState.load(state.run_id, tmp_path) + assert loaded.step_results["gate"]["output"]["message"] == "2026-01-01" + + +def test_fan_out_worker_exception_stops_dispatch_before_earlier_items_finish( + tmp_path, monkeypatch +): + from types import SimpleNamespace + + from specify_cli.workflows import _execution + + halted_seen = threading.Event() + + class ObservedEvent(threading.Event): + def set(self): + super().set() + halted_seen.set() + + monkeypatch.setattr(_execution, "threading", SimpleNamespace(Event=ObservedEvent)) + started = set() + lock = threading.Lock() + + class Blow(StepBase): + type_key = "blow" + + def execute(self, config, context): + with lock: + started.add(context.item) + if context.item == 2: + raise RuntimeError("boom") + if context.item in {0, 1}: + # Earlier items finish only after the failure halts dispatch. + halted_seen.wait(2) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "blow", Blow()) + with pytest.raises(RuntimeError, match="boom"): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2, 3, 4, 5], + "max_concurrency": 3, + "step": {"id": "item", "type": "blow"}, + } + ], + ) + ) + + assert started == {0, 1, 2} From 1b0a3a46ad22df04e3d5d1c2666fb00d481bedd4 Mon Sep 17 00:00:00 2001 From: Markus Date: Tue, 29 Sep 2026 21:30:04 +0200 Subject: [PATCH 23/28] fix(workflows): persist the active step before it executes The executor updated current_step_id and current_step_index only in memory and wrote them with the next checkpoint, after the step had finished. While a long step ran, and after a hard crash, state.json still named the previous step, so `workflow status` and the run payload reported the wrong occurrence. main saves before each step starts. Checkpoint the active occurrence before it executes, so root steps, later loop iterations, and fan-out items are persisted under their qualified IDs while they run. This adds one state write per started step, as on main; the checkpoint-count tests are updated accordingly. Assisted-by: OpenCode (model: claude-opus-5.5, autonomous) --- src/specify_cli/workflows/_execution.py | 3 + tests/workflows/test_composition_execution.py | 63 ++++++++++++++++++- workflows/ARCHITECTURE.md | 6 +- 3 files changed, 68 insertions(+), 4 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 86e704cbd6..e95813141c 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -474,6 +474,9 @@ def step(self, config, node, context, ancestry, path, public, qualified): if node["phase"] in {"ready", "blocked"}: with self.state._lock: self.state.current_step_id = qualified + # Persist the active occurrence before it runs, as on main, so status + # reports it for the whole duration of a long step and after a crash. + self.commit() self.emit( "step_started", qualified, diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 20f7f17bef..e9bb7764de 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -2347,7 +2347,9 @@ def save(self): ) assert state.status == RunStatus.COMPLETED - assert saves == items + 5 + # Each started occurrence (fan and items) is checkpointed before it runs, + # and each item result once more before it is done. + assert saves == 2 * items + 6 def test_fan_out_snapshot_size_does_not_scale_with_template_length(tmp_path, probe): @@ -2430,7 +2432,8 @@ def save(self): state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) assert state.status == RunStatus.COMPLETED - assert saves == 3 + # Restarting ``wait`` persists it as active before it runs, then its result. + assert saves == 4 def test_exact_depth_limit_is_allowed(tmp_path, probe): @@ -2605,6 +2608,62 @@ def execute(self, config, context): assert RunState.load(state.run_id, tmp_path).current_step_id == "loop:body:1" +@pytest.mark.parametrize( + ("steps", "expected"), + [ + ( + [{"id": "first", "type": "probe"}, {"id": "observe", "type": "observe"}], + [("observe", 1)], + ), + ( + [ + {"id": "first", "type": "probe"}, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "observe", "type": "observe"}], + }, + ], + [("observe", 1), ("loop:observe:1", 1)], + ), + ( + [ + {"id": "first", "type": "probe"}, + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "observe", "type": "observe"}, + }, + ], + [("fan:observe:0", 1), ("fan:observe:1", 1)], + ), + ], + ids=["root", "later-loop-iteration", "fan-out-item"], +) +def test_running_step_is_persisted_before_it_executes( + tmp_path, monkeypatch, probe, steps, expected +): + seen = [] + + class Observe(StepBase): + type_key = "observe" + + def execute(self, config, context): + persisted = RunState.load(context.run_id, tmp_path) + seen.append((persisted.current_step_id, persisted.current_step_index)) + return StepResult(output={}) + + monkeypatch.setitem(STEP_REGISTRY, "observe", Observe()) + + state = WorkflowEngine(tmp_path).execute(definition("parent", steps)) + + assert state.status == RunStatus.COMPLETED + assert seen == expected + + def test_failed_fan_out_item_exception_reports_qualified_current_step_id( tmp_path, monkeypatch ): diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index 8e5397fc2a..a965a2cd78 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -79,8 +79,10 @@ New runs use a versioned execution tree. Each occurrence owns its result, selected child sequences, and optional workflow binding. A binding stores the target, frozen definition, private inputs, and `workflow_dir`; the called workflow remains a private scope in the same run. Fan-out items have separate -contexts. Binding and selected expansions are checkpointed before child side -effects, results before an occurrence is done, and logs after the checkpoint. +contexts. Each occurrence is checkpointed as the active step before it executes, +so status reports it while it runs. Binding and selected expansions are +checkpointed before child side effects, results before an occurrence is done, +and logs after the checkpoint. Legacy runs enter through their top-level index once. Inputs and tree transitions share one atomic state checkpoint; the inputs file is a compatibility mirror. A checkpoint failure prevents further writes by that From 3e5e66e882164f3df9832748c35b7c54816b4bd7 Mon Sep 17 00:00:00 2001 From: Markus Date: Wed, 30 Sep 2026 18:28:19 +0200 Subject: [PATCH 24/28] fix(workflows): persist workflow calls before announcing them 1b0a3a46 checkpointed the active occurrence before step_started, but only on the shared step path. Workflow calls return to Execution.workflow() before that path and emitted step_started without a checkpoint. Until binding was saved, and on resume of an already-bound call until the first child step, state.json named the previous step, and a crash or checkpoint failure could leave a start event for the call that no checkpoint reflected. Move the start sequence (set current_step_id, checkpoint, then log and callback) into one Execution.start() helper used by both paths, so the rule lives in one place. The ID update and checkpoint now happen under one lock hold. Tests cover top-level, nested, and resumed bound calls, and a failed start checkpoint that must leave no start event; all three failed before this change. Assisted-by: OpenCode (model: claude-opus-5.5, autonomous) --- src/specify_cli/workflows/_execution.py | 43 +++---- tests/workflows/test_composition_execution.py | 110 ++++++++++++++++++ workflows/ARCHITECTURE.md | 5 +- 3 files changed, 135 insertions(+), 23 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index e95813141c..c0845c1d6f 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -380,6 +380,24 @@ def emit( with self.engine._callback_lock: self.engine.on_step_start(qualified, callback_label) + def start(self, config, qualified, path, ancestry): + """Persist an occurrence as the active step, then announce its start. + + Every step path starts here, so the checkpoint always precedes the + ``step_started`` event and callback, as on main: status reports the + occurrence while it runs and after a crash, and a failed checkpoint + leaves no start event behind. + """ + kind = config.get("type", "command") + with self.state._lock: + self.state.current_step_id = qualified + self.commit() + # A call is labelled by its type; a stray ``command`` key must not rename it. + label = kind if kind == "workflow" else config.get("command", "") or kind + self.emit( + "step_started", qualified, path, ancestry, type=kind, callback_label=label + ) + def run( self, seq, @@ -472,19 +490,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): ) if node["phase"] in {"ready", "blocked"}: - with self.state._lock: - self.state.current_step_id = qualified - # Persist the active occurrence before it runs, as on main, so status - # reports it for the whole duration of a long step and after a crash. - self.commit() - self.emit( - "step_started", - qualified, - path, - ancestry, - type=kind, - callback_label=config.get("command", "") or kind, - ) + self.start(config, qualified, path, ancestry) impl = self.registry.get(kind) if impl is None: # As on main: terminal, only step_failed, no projected result. @@ -697,14 +703,9 @@ def workflow( ): from .engine import WorkflowDefinition, workflow_dir_for - self.emit( - "step_started", - qualified, - path, - ancestry, - type="workflow", - callback_label="workflow", - ) + # Unlike other steps, a call is announced on every entry, including a + # resumed call that is already bound and skips the binding checkpoint. + self.start(config, qualified, path, ancestry) binding = node.get("binding") target = binding["workflow"] if binding else config.get("workflow") if binding is None: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index e9bb7764de..17df906ff8 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -2664,6 +2664,116 @@ def execute(self, config, context): assert seen == expected +def _record_persisted_step_on_start(engine, tmp_path, seen): + """Record the persisted ``current_step_id`` whenever a step start is announced.""" + + def on_step_start(step_id, label): + (path,) = (tmp_path / ".specify/workflows/runs").glob("*/state.json") + persisted = json.loads(path.read_text(encoding="utf-8"))["current_step_id"] + seen.append((step_id, label, persisted)) + + engine.on_step_start = on_step_start + return engine + + +def test_workflow_call_is_persisted_before_it_starts(tmp_path, probe): + install(tmp_path, definition("grandchild", [{"id": "leaf", "type": "probe"}])) + install( + tmp_path, + definition( + "child", + [ + {"id": "prepare", "type": "probe"}, + {"id": "nested", "type": "workflow", "workflow": "grandchild"}, + ], + ), + ) + seen = [] + engine = _record_persisted_step_on_start(WorkflowEngine(tmp_path), tmp_path, seen) + + state = engine.execute(definition("parent", [{"id": "first", "type": "probe"}, call()])) + + assert state.status == RunStatus.COMPLETED + # Like every other step, a call (top-level and nested inside a called + # workflow, by its workflow-relative ID) is checkpointed as the active step + # before its start is logged and announced. + assert seen == [ + ("first", "probe", "first"), + ("call", "workflow", "call"), + ("prepare", "probe", "prepare"), + ("nested", "workflow", "nested"), + ("leaf", "probe", "leaf"), + ] + + +def test_resumed_bound_workflow_call_is_persisted_before_it_starts(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).current_step_id == "wait" + seen = [] + engine = _record_persisted_step_on_start(WorkflowEngine(tmp_path), tmp_path, seen) + + state = engine.resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + # The already-bound call skips binding, which previously held the only + # pre-child checkpoint; it must still persist itself before re-announcing. + assert seen == [("call", "workflow", "call"), ("wait", "probe", "wait")] + + +def test_workflow_call_start_checkpoint_failure_logs_no_start(tmp_path, monkeypatch, probe): + from specify_cli.workflows._execution import CheckpointError + + install(tmp_path, definition("child", [{"id": "inner", "type": "probe"}])) + original = RunState._atomic_write_json + + def write(path, data): + if path.name == "state.json" and data.get("current_step_id") == "call": + raise OSError("checkpoint failure") + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, label: callbacks.append(step_id) + + with pytest.raises(CheckpointError, match="checkpoint failure"): + engine.execute( + definition("parent", [{"id": "first", "type": "probe"}, call()]), + run_id="fault", + ) + + runs = tmp_path / ".specify/workflows/runs/fault" + started = [ + entry["step_id"] + for entry in map(json.loads, (runs / "log.jsonl").read_text().splitlines()) + if entry["event"] == "step_started" + ] + disk = json.loads((runs / "state.json").read_text()) + node = disk["execution"]["sequence"]["nodes"][1] + # The failed start checkpoint is not followed by a start event, a callback, + # or target binding. + assert started == ["first"] + assert callbacks == ["first"] + assert disk["current_step_id"] == "first" + assert node["phase"] == "ready" + assert "binding" not in node + assert probe["inner"] == 0 + + def test_failed_fan_out_item_exception_reports_qualified_current_step_id( tmp_path, monkeypatch ): diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index a965a2cd78..9f559ce981 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -79,8 +79,9 @@ New runs use a versioned execution tree. Each occurrence owns its result, selected child sequences, and optional workflow binding. A binding stores the target, frozen definition, private inputs, and `workflow_dir`; the called workflow remains a private scope in the same run. Fan-out items have separate -contexts. Each occurrence is checkpointed as the active step before it executes, -so status reports it while it runs. Binding and selected expansions are +contexts. Each occurrence, including a workflow call, is checkpointed as the +active step before its start is logged and before it executes, so status +reports it while it runs. Binding and selected expansions are checkpointed before child side effects, results before an occurrence is done, and logs after the checkpoint. Legacy runs enter through their top-level index once. Inputs and tree From e5a4223bc6c2fd141a397adcabbed22b851e6bbd Mon Sep 17 00:00:00 2001 From: Markus Date: Wed, 30 Sep 2026 18:44:17 +0200 Subject: [PATCH 25/28] fix(workflows): let interrupts during checkpoints pause the run RunState.save() caught BaseException around the checkpoint write, so a KeyboardInterrupt during any checkpoint became CheckpointError, marked the state as failed to checkpoint, and bypassed the KeyboardInterrupt handler in execute()/resume(). The run stayed "running" and could not be resumed. On main the interrupt reaches that handler and the run pauses. Introduced in 7aa424dd. Convert only Exception to CheckpointError. Write failures still stop further writes; interrupts propagate unchanged, and the pause re-saves the tree from memory. Tests interrupt a checkpoint at the root, inside a workflow call, and during resume; all failed before this change. Assisted-by: OpenCode (model: claude-opus-5.5, autonomous) --- src/specify_cli/workflows/engine.py | 5 +- tests/workflows/test_composition_execution.py | 80 +++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-) diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 93ba771cb1..384fdbbb79 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -774,7 +774,10 @@ def save(self) -> None: try: self._atomic_write_json(runs_dir / "state.json", state_data) self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs}) - except BaseException as exc: + except Exception as exc: + # Only write failures poison the instance. A graceful interrupt + # (KeyboardInterrupt) propagates unchanged so execute()/resume() + # pause the run, as on main; the pause re-saves from memory. from ._execution import CheckpointError self._checkpoint_failed = True diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 17df906ff8..8ce146e204 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -681,6 +681,86 @@ def write(path, data): assert probe["work"] == 1 +def _interrupt_state_write(monkeypatch, when): + """Raise ``KeyboardInterrupt`` from the first ``state.json`` write matching *when*.""" + original = RunState._atomic_write_json + fired = [] + + def write(path, data): + if path.name == "state.json" and not fired and when(data): + fired.append(path) + raise KeyboardInterrupt + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + return fired + + +def _logged_events(state): + log = (state.runs_dir / "log.jsonl").read_text(encoding="utf-8").splitlines() + return [json.loads(line)["event"] for line in log] + + +@pytest.mark.parametrize("scope", ["root", "workflow-call"]) +def test_interrupt_during_checkpoint_pauses_run(tmp_path, monkeypatch, probe, scope): + steps = [{"id": "first", "type": "probe"}, {"id": "second", "type": "probe"}] + if scope == "workflow-call": + install(tmp_path, definition("child", steps)) + steps = [call()] + # Interrupt the checkpoint that marks ``second`` active, after ``first`` ran. + fired = _interrupt_state_write( + monkeypatch, lambda data: data.get("current_step_id") == "second" + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", steps)) + + # A graceful interrupt is not a checkpoint failure: it reaches the engine's + # pause path, as on main, even inside a called workflow. + assert fired + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + assert probe == {"first": 1} + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + assert probe == {"first": 1, "second": 1} + + +def test_interrupt_during_resume_checkpoint_pauses_run(tmp_path, monkeypatch, probe): + root = definition( + "parent", + [ + {"id": "wait", "type": "probe", "await": True}, + {"id": "next", "type": "probe"}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + # Interrupt the checkpoint that records ``wait`` as completed on resume. + fired = _interrupt_state_write( + monkeypatch, + lambda data: data.get("step_results", {}).get("wait", {}).get("status") + == "completed", + ) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert fired + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + # The interrupted checkpoint's transition is saved by the pause, so the + # completed ``wait`` is not run a third time. + assert state.status == RunStatus.COMPLETED + assert probe == {"wait": 2, "next": 1} + + def test_checkpoint_failure_stops_concurrent_fan_out_logging(tmp_path, monkeypatch): import specify_cli.workflows._execution as execution from specify_cli.workflows._execution import CheckpointError From 8e9c5fa46cbf2d98a1c680baa609b8fa3e8b5fcc Mon Sep 17 00:00:00 2001 From: Markus Date: Wed, 30 Sep 2026 20:21:31 +0200 Subject: [PATCH 26/28] fix(workflows): replay the result of aborted fan-out items 6792bea1 made fan-out items publish only the result their traversal projected, so missing step implementations publish nothing. The replay of an already-aborted occurrence returned without projecting its stored result, so after a sibling item resumed, the aborted item appeared as {} in the fan-out's ordered results. Project the stored result before returning "aborted", as the live run did when the occurrence aborted. Nothing is re-executed and no events are emitted. The aborted-sibling regression now checks the item output for a step, an if container, and a workflow call template; all three failed before this change. Assisted-by: OpenCode (model: claude-opus-5.5, autonomous) --- src/specify_cli/workflows/_execution.py | 6 +++ tests/workflows/test_composition_execution.py | 41 +++++++++++++++++-- 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index c0845c1d6f..5627f9d69f 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -476,6 +476,12 @@ def step(self, config, node, context, ancestry, path, public, qualified): ) return "completed" if node.get("outcome") == "aborted": + # Replay the result the aborted occurrence projected when it ran, so + # its fan-out item keeps its published output (see run_item). + if node.get("result") is not None: + self.project( + context, name, node["result"], public=public, qualified=qualified + ) return "aborted" if kind == "workflow": diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 8ce146e204..66442d02b3 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -2532,7 +2532,21 @@ def test_exact_depth_limit_is_allowed(tmp_path, probe): assert probe["work"] == 1 -def test_aborted_fanout_sibling_is_never_restarted(tmp_path, monkeypatch): +@pytest.mark.parametrize( + "template", + [ + {"id": "mixed", "type": "mixed"}, + { + "id": "mixed", + "type": "if", + "condition": True, + "then": [{"id": "inner", "type": "mixed"}], + }, + call("child", id="mixed", input={"n": "{{ item }}"}), + ], + ids=["step", "if-container", "workflow-call"], +) +def test_aborted_fanout_sibling_is_never_restarted(tmp_path, monkeypatch, template): barrier = threading.Barrier(2, timeout=5) counts = Counter() @@ -2540,16 +2554,25 @@ class Mixed(StepBase): type_key = "mixed" def execute(self, config, context): - counts[context.item] += 1 + item = context.item if context.item is not None else context.inputs["n"] + counts[item] += 1 if not context.is_resume: barrier.wait() - if context.item == 0: + if item == 0: return StepResult( StepStatus.COMPLETED if context.is_resume else StepStatus.PAUSED ) return StepResult(StepStatus.FAILED, output={"aborted": True}) monkeypatch.setitem(STEP_REGISTRY, "mixed", Mixed()) + install( + tmp_path, + definition( + "child", + [{"id": "inner", "type": "mixed"}], + inputs={"n": {"type": "number"}}, + ), + ) root = definition( "parent", [ @@ -2558,7 +2581,7 @@ def execute(self, config, context): "type": "fan-out", "items": [0, 1], "max_concurrency": 2, - "step": {"id": "mixed", "type": "mixed"}, + "step": template, } ], ) @@ -2567,6 +2590,16 @@ def execute(self, config, context): state = WorkflowEngine(tmp_path).resume(state.run_id) assert state.status == RunStatus.ABORTED assert counts == {0: 2, 1: 1} + # Replaying the aborted item keeps the result it published when it ran. + published = state.step_results["spread:mixed:1"]["output"] + assert published + for results in ( + state.step_results["spread"]["output"]["results"], + RunState.load(state.run_id, tmp_path).step_results["spread"]["output"][ + "results" + ], + ): + assert results[1] == published NAN_SNAPSHOT = """ From 4db5a079da671048fc5c64a76a72a859fa5725dd Mon Sep 17 00:00:00 2001 From: Markus Date: Wed, 30 Sep 2026 20:42:10 +0200 Subject: [PATCH 27/28] fix(workflows): hide partial fan-out results from resumed items A halted fan-out checkpoints its partial results in its record for reporting. On resume that record is projected before the items restart, so re-executed and not-yet-started items saw steps..output.results with a stale prefix, including the paused item's own earlier output. Items in a live run, and on main, never see results: the engine adds them only after the items finish. Present since 7aa424dd. Remove the engine-added results from the fan-out's own entry in the item-local snapshot. The parent context and checkpoint keep them. Add an invariant test: a resumed item must see exactly the context it has in a run that never paused, for step, if-container, and workflow-call templates, sequentially and in parallel. It found only this difference; the four affected cases failed before this change. Assisted-by: OpenCode (model: claude-opus-5.5, autonomous) --- src/specify_cli/workflows/_execution.py | 8 ++ tests/workflows/test_composition_execution.py | 120 ++++++++++++++++++ 2 files changed, 128 insertions(+) diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 5627f9d69f..4ada1ad34b 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -821,6 +821,14 @@ def fan_out( workers = 1 workers = min(workers, len(items)) initial = deepcopy(context.steps) + # Items run before the engine adds ``results`` to the fan-out record. A + # resumed fan-out's checkpointed record carries the partial results, so + # hide them from item contexts; the reporting view keeps them. + for key in {config.get("id", "step-0"), qualified}: + record = initial.get(key) + if isinstance(record, dict) and "results" in record.get("output", {}): + output = {k: v for k, v in record["output"].items() if k != "results"} + initial[key] = {**record, "output": output} halted = threading.Event() template_name = child_steps(config, node, 0)[0]["id"] diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index 66442d02b3..c8f0250450 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -2602,6 +2602,126 @@ def execute(self, config, context): assert results[1] == published +FAN_OUT_ITEM_TEMPLATES = [ + {"id": "tmpl", "type": "snap"}, + { + "id": "tmpl", + "type": "if", + "condition": True, + "then": [{"id": "inner", "type": "snap"}], + }, + call("child", id="tmpl", input={"n": "{{ item }}"}), +] +FAN_OUT_ITEM_TEMPLATE_IDS = ["step", "if-container", "workflow-call"] + + +@pytest.mark.parametrize("max_concurrency", [1, 3], ids=["sequential", "parallel"]) +@pytest.mark.parametrize( + "template", FAN_OUT_ITEM_TEMPLATES, ids=FAN_OUT_ITEM_TEMPLATE_IDS +) +def test_resumed_fan_out_item_sees_its_uninterrupted_context( + tmp_path, monkeypatch, probe, template, max_concurrency +): + """Exact resume must re-run a fan-out item in the context it has in a run + that never paused. Replay reconstructs item contexts from the checkpoint, + so any difference (for example, a result the live run adds only after its + items) is a live/replay gap.""" + from dataclasses import asdict + + pause = {"enabled": False} + seen = {} + + class Snap(StepBase): + type_key = "snap" + + def execute(self, config, context): + item = context.item if context.item is not None else context.inputs["n"] + view = asdict(context) + del view["run_id"], view["is_resume"] + phase = "resume" if context.is_resume else pause["enabled"] + seen.setdefault(phase, {})[item] = view + if item == 1 and pause["enabled"] and not context.is_resume: + return StepResult(StepStatus.PAUSED, output={"item": item}) + return StepResult(output={"item": item}) + + monkeypatch.setitem(STEP_REGISTRY, "snap", Snap()) + install( + tmp_path, + definition( + "child", + [{"id": "inner", "type": "snap"}], + inputs={"n": {"type": "number"}}, + ), + ) + root = definition( + "parent", + [ + {"id": "setup", "type": "probe", "value": "ready"}, + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2], + "max_concurrency": max_concurrency, + "step": template, + }, + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.COMPLETED + pause["enabled"] = True + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + resumed = seen["resume"] + assert 1 in resumed + for item, view in resumed.items(): + assert view == seen[False][item], f"item {item} context differs on resume" + + +def test_resumed_fan_out_items_do_not_see_partial_results(tmp_path, monkeypatch): + seen = [] + + class Look(StepBase): + type_key = "look" + + def execute(self, config, context): + seen.append("results" in context.steps["fan"]["output"]) + if context.item == 1 and not context.is_resume: + return StepResult(StepStatus.PAUSED, output={"item": 1}) + return StepResult(output={"item": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "look", Look()) + root = definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2], + "step": {"id": "look", "type": "look"}, + } + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + # The paused fan-out still reports its partial results. + assert state.step_results["fan"]["output"]["results"] == [{"item": 0}, {"item": 1}] + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + # Items never see the engine-added results: not live, not after resume. + assert seen == [False, False, False, False] + assert state.step_results["fan"]["output"]["results"] == [ + {"item": 0}, + {"item": 1}, + {"item": 2}, + ] + + NAN_SNAPSHOT = """ schema_version: "1.0" workflow: From c049f698e49675fd984504d74dc3fd306adc1a88 Mon Sep 17 00:00:00 2001 From: Markus Date: Thu, 1 Oct 2026 17:54:57 +0200 Subject: [PATCH 28/28] fix(workflows): validate occurrence lifecycle and report interrupted scopes Persist fan-out templates as YAML, reject invalid resume offsets and tree states, and check only changed nodes during lifecycle transitions. Preserve original step exceptions during best-effort cleanup and derive interrupted bound-call status from the run. Cover regressions and document the contract. Assisted-by: OpenCode (model: gpt-6-sol, autonomous) --- design/workflow-step.md | 6 + src/specify_cli/workflows/_commands.py | 2 +- src/specify_cli/workflows/_execution.py | 699 ++++++++++-------- src/specify_cli/workflows/command_status.py | 2 +- src/specify_cli/workflows/engine.py | 11 +- .../workflows/test_command_status.py | 49 ++ tests/workflows/test_composition_execution.py | 290 +++++++- workflows/ARCHITECTURE.md | 53 ++ 8 files changed, 766 insertions(+), 346 deletions(-) diff --git a/design/workflow-step.md b/design/workflow-step.md index 9011b3de8d..cef0c1b903 100644 --- a/design/workflow-step.md +++ b/design/workflow-step.md @@ -51,6 +51,12 @@ frozen across resume. This is intentionally forward-only: new resume inputs do not reinterpret or repeat completed work. Legacy states enter the tree once through their saved top-level index. +The common occurrence runner owns start, checkpoint, replay, and finalization. +Step implementations return results or expansions; they do not manage persisted +lifecycle state. Internal workflow-call and registered-step paths share the +same checked transitions and post-checkpoint notifications. Per-occurrence +activity belongs to the execution tree, never to the shared step instance. + Fan-out items have independent expression contexts. Their internal results do not enter the shared parent context; the parent receives the qualified item result and the fan-out step's ordered `output.results`. Fan-out item aliases are diff --git a/src/specify_cli/workflows/_commands.py b/src/specify_cli/workflows/_commands.py index a186abf8b4..925456f021 100644 --- a/src/specify_cli/workflows/_commands.py +++ b/src/specify_cli/workflows/_commands.py @@ -950,7 +950,7 @@ def _workflow_run_payload(state: Any) -> dict[str, Any]: if getattr(state, "execution", None): from ._execution import scope_summaries - scopes = scope_summaries(state.execution) + scopes = scope_summaries(state.execution, state.status.value) if scopes: payload["workflow_scopes"] = scopes return payload diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py index 4ada1ad34b..f4a100142d 100644 --- a/src/specify_cli/workflows/_execution.py +++ b/src/specify_cli/workflows/_execution.py @@ -8,7 +8,7 @@ from concurrent.futures import ThreadPoolExecutor from copy import deepcopy -from dataclasses import replace +from dataclasses import dataclass, replace import threading from typing import Any @@ -25,6 +25,7 @@ from .expressions import evaluate_condition, evaluate_expression HALTING = {"paused", "failed", "aborted"} +EXECUTION_VERSION = 2 def unknown_step_error(kind): return f"Unknown step type: {kind!r}" @@ -51,7 +52,7 @@ def new_execution( ) -> dict[str, Any]: """Create the persisted root execution tree.""" return { - "version": 1, + "version": EXECUTION_VERSION, "sequence": sequence(steps), "offset": max(0, offset), "initial": deepcopy(initial), @@ -64,6 +65,17 @@ def steps_of(seq: dict[str, Any]) -> Any: LOOP_TYPES = frozenset({"while", "do-while"}) +TRANSITIONS = { + "begin": (frozenset({"ready", "blocked", "children", "outputs"}), frozenset()), + "expand": (frozenset({"ready", "blocked"}), frozenset({"result", "children", "template"})), + "bind": (frozenset({"ready", "blocked"}), frozenset({"binding", "children"})), + "rebind": (frozenset({"children", "blocked", "outputs"}), frozenset({"binding"})), + "iterate": (frozenset({"children"}), frozenset({"children"})), + "outputs": (frozenset({"children", "blocked", "outputs"}), frozenset()), + "finish": (frozenset({"ready", "blocked", "children", "outputs"}), frozenset({"result", "outcome", "error"})), + "settle": (frozenset({"children"}), frozenset({"outcome", "error", "fan_results"})), + "leave": (frozenset({"ready", "blocked", "children", "outputs", "done"}), frozenset()), +} def shares_source(kind, index): @@ -79,7 +91,7 @@ def child_steps(config, node, index): if kind == "workflow": return yaml.safe_load(node["binding"]["definition"])["steps"] if kind == "fan-out": - template = node["result"]["output"]["step_template"] + template = yaml.safe_load(node["template"]) return [{"id": "item", **template}] return steps_of(node["children"][0]) @@ -134,120 +146,161 @@ def walk_execution( yield config, node, here, qualified -def validate_execution(tree: Any) -> None: - """Validate stored structure without importing a project's custom steps.""" - - def check_sequence(seq, steps=None, *, shared=False): - if not isinstance(seq, dict) or (shared and "source" in seq): +def check_sequence(seq, steps=None, *, shared=False): + """Walk a stored sequence, checking every occurrence and its descendants.""" + if not isinstance(seq, dict) or (shared and "source" in seq): + raise ValueError("Invalid execution sequence") + if shared: + if not isinstance(steps, list): + raise ValueError("Invalid shared execution source") + else: + if not isinstance(seq.get("source"), str): raise ValueError("Invalid execution sequence") - if shared: - if not isinstance(steps, list): - raise ValueError("Invalid shared execution source") - else: - if not isinstance(seq.get("source"), str): - raise ValueError("Invalid execution sequence") - steps = steps_of(seq) - nodes = seq.get("nodes") - if ( - not isinstance(steps, list) - or not all(isinstance(s, dict) for s in steps) - or not isinstance(nodes, list) - or len(nodes) != len(steps) - ): - raise ValueError("Invalid execution sequence length or steps") - for step, node in zip(steps, nodes): - if not isinstance(node, dict) or node.get("phase") not in { - "ready", - "children", - "outputs", - "blocked", - "done", - }: - raise ValueError("Invalid execution phase") - result = node.get("result") - if result is not None and ( - not isinstance(result, dict) - or result.get("status") not in {s.value for s in StepStatus} - or not isinstance(result.get("output"), dict) - ): - raise ValueError("Invalid execution result") - if node["phase"] == "done" and result is None: - raise ValueError("Completed execution lacks a result") - if node.get("outcome", "completed") not in {"completed", *HALTING}: - raise ValueError("Invalid execution outcome") + steps = steps_of(seq) + nodes = seq.get("nodes") + if ( + not isinstance(steps, list) + or not all(isinstance(s, dict) for s in steps) + or not isinstance(nodes, list) + or len(nodes) != len(steps) + ): + raise ValueError("Invalid execution sequence length or steps") + for step, node in zip(steps, nodes): + check_node(step, node) + for index, child in enumerate(node.get("children", [])): + if shares_source(step.get("type"), index): + check_sequence(child, child_steps(step, node, index), shared=True) + else: + check_sequence(child) + + +def check_node(step, node, *, changed=None, new_children=()): + """Validate one occurrence and its direct children without walking descendants. + + On a transition, unchanged YAML sources and existing child shapes were + checked when they were created or loaded. Load checks everything. + """ + if not isinstance(node, dict) or node.get("phase") not in { + "ready", "children", "outputs", "blocked", "done", + }: + raise ValueError("Invalid execution phase") + result = node.get("result") + if result is not None and ( + not isinstance(result, dict) + or result.get("status") not in {s.value for s in StepStatus} + or not isinstance(result.get("output"), dict) + ): + raise ValueError("Invalid execution result") + if node["phase"] == "done" and result is None: + raise ValueError("Completed execution lacks a result") + if "active" in node and type(node["active"]) is not bool: + raise ValueError("Invalid execution activity") + if node["phase"] == "done" and node.get("active"): + raise ValueError("Completed execution is still active") + if node.get("outcome", "completed") not in {"completed", *HALTING}: + raise ValueError("Invalid execution outcome") + if node["phase"] == "done" and node.get("outcome", "completed") != "completed": + raise ValueError("Completed execution has a blocking outcome") + if node["phase"] == "blocked" and ( + result is None or result["status"] not in {"failed", "paused"} + or node.get("outcome") not in HALTING + ): + raise ValueError("Blocked execution lacks a blocking result") + children = node.get("children", []) + if not isinstance(children, list): + raise ValueError("Invalid execution children") + kind = step.get("type") + if "fan_results" in node and ( + kind != "fan-out" or not isinstance(node["fan_results"], list) + ): + raise ValueError("Invalid fan-out aggregate results") + if "template" in node and (kind != "fan-out" or not isinstance(node["template"], str)): + raise ValueError("Invalid fan-out template") + binding = node.get("binding") + if binding is not None: + if kind != "workflow" or not isinstance(binding, dict): + raise ValueError("Invalid workflow binding") + if changed is None or "binding" in changed: + source = binding.get("definition") + if not isinstance(source, str): + raise ValueError("Invalid bound workflow definition or inputs") + bound_definition = yaml.safe_load(source) if ( - node["phase"] == "done" - and node.get("outcome", "completed") != "completed" - ): - raise ValueError("Completed execution has a blocking outcome") - if node["phase"] == "blocked" and ( - result is None - or result["status"] not in {"failed", "paused"} - or node.get("outcome") not in HALTING + not isinstance(bound_definition, dict) + or not isinstance(bound_definition.get("workflow"), dict) + or bound_definition["workflow"].get("id") != binding.get("workflow") + or not isinstance(binding.get("inputs"), dict) + or "workflow_dir" not in binding + or not isinstance(binding["workflow_dir"], (str, type(None))) + or len(children) != 1 + or not isinstance(bound_definition.get("steps"), list) ): - raise ValueError("Blocked execution lacks a blocking result") - children = node.get("children", []) - if not isinstance(children, list): - raise ValueError("Invalid execution children") - kind = step.get("type") - binding = node.get("binding") - if binding is not None: - if kind != "workflow" or not isinstance(binding, dict): - raise ValueError("Invalid workflow binding") - source = binding.get("definition") - if not isinstance(source, str): - raise ValueError("Invalid bound workflow definition or inputs") - definition = yaml.safe_load(source) - if ( - not isinstance(definition, dict) - or not isinstance(definition.get("workflow"), dict) - or definition["workflow"].get("id") != binding.get("workflow") - or not isinstance(binding.get("inputs"), dict) - or "workflow_dir" not in binding - or not isinstance(binding["workflow_dir"], (str, type(None))) - or len(children) != 1 - or not isinstance(definition.get("steps"), list) - ): - raise ValueError("Invalid bound workflow definition or inputs") - elif kind == "workflow" and children: - raise ValueError("Workflow children require a binding") - if kind == "fan-out" and children: - template = (result or {}).get("output", {}).get("step_template") - if not isinstance(template, dict): - raise ValueError("Invalid fan-out template") - for index, child in enumerate(children): - if shares_source(kind, index): - check_sequence(child, child_steps(step, node, index), shared=True) - else: - check_sequence(child) - handled_child_failure = ( - kind == "workflow" - and step.get("continue_on_error") is True - and node.get("outcome") == "completed" - and result is not None - and result["status"] == "failed" - ) - if ( - node["phase"] == "done" - and any(nested["phase"] != "done" for nested in child["nodes"]) - and not handled_child_failure - ): - raise ValueError("Completed execution has unfinished children") - if node["phase"] == "outputs" and binding is None: - raise ValueError("Output finalization requires a workflow binding") - if node["phase"] == "children" and ( - not children or (binding is None and result is None) - ): - raise ValueError("Expanded execution lacks its children or result") - if node["phase"] == "ready" and (result is not None or children or binding): - raise ValueError("Unstarted execution already has progress") - if step.get("type") == "fan-out" and children: - items = (result or {}).get("output", {}).get("items") - if not isinstance(items, list) or len(items) != len(children): - raise ValueError("Fan-out items do not match execution children") + raise ValueError("Invalid bound workflow definition or inputs") + elif kind == "workflow" and children: + raise ValueError("Workflow children require a binding") + if kind == "fan-out" and children and (changed is None or "template" in changed): + source = node.get("template") + template = yaml.safe_load(source) if isinstance(source, str) else None + if not isinstance(template, dict): + raise ValueError("Invalid fan-out template") + if changed is None or new_children: + shared_steps = None + for index, child in enumerate(children): + if changed is not None and not any(child is fresh for fresh in new_children): + continue + if shares_source(kind, index): + if shared_steps is None: + shared_steps = child_steps(step, node, index) + check_sequence_shape(child, shared_steps, shared=True) + else: + check_sequence_shape(child) + handled_child_failure = ( + kind == "workflow" and step.get("continue_on_error") is True + and node.get("outcome") == "completed" and result is not None + and result["status"] == "failed" + ) + if node["phase"] == "done" and not handled_child_failure: + for child in children: + if any(nested["phase"] != "done" for nested in child["nodes"]): + raise ValueError("Completed execution has unfinished children") + if node["phase"] == "outputs" and binding is None: + raise ValueError("Output finalization requires a workflow binding") + if node["phase"] == "children" and ( + not children or (binding is None and result is None) + ): + raise ValueError("Expanded execution lacks its children or result") + if node["phase"] == "ready" and (result is not None or children or binding): + raise ValueError("Unstarted execution already has progress") + if kind == "fan-out" and children: + items = (result or {}).get("output", {}).get("items") + if not isinstance(items, list) or len(items) != len(children): + raise ValueError("Fan-out items do not match execution children") + + +def check_sequence_shape(seq, steps=None, *, shared=False): + """Check a newly attached child sequence without visiting its nodes' descendants.""" + if not isinstance(seq, dict) or (shared and "source" in seq): + raise ValueError("Invalid execution sequence") + if shared: + if not isinstance(steps, list): + raise ValueError("Invalid shared execution source") + else: + if not isinstance(seq.get("source"), str): + raise ValueError("Invalid execution sequence") + steps = steps_of(seq) + nodes = seq.get("nodes") + if ( + not isinstance(steps, list) or not all(isinstance(s, dict) for s in steps) + or not isinstance(nodes, list) or len(nodes) != len(steps) + ): + raise ValueError("Invalid execution sequence length or steps") + + +def validate_execution(tree: Any, *, workflow_steps=None, current_step_index=None) -> None: + """Validate stored structure without importing a project's custom steps.""" try: - if not isinstance(tree, dict) or tree.get("version") != 1: + if not isinstance(tree, dict) or tree.get("version") != EXECUTION_VERSION: raise ValueError("Unsupported execution version") offset = tree.get("offset", 0) if ( @@ -257,6 +310,11 @@ def check_sequence(seq, steps=None, *, shared=False): ): raise ValueError("Invalid execution offset or initial aliases") check_sequence(tree["sequence"]) + if workflow_steps is not None: + if offset >= len(workflow_steps) or offset > current_step_index: + raise ValueError("Invalid execution offset for workflow position") + if steps_of(tree["sequence"]) != workflow_steps[offset:]: + raise ValueError("Invalid execution state: root sequence differs from workflow snapshot") except (KeyError, TypeError, yaml.YAMLError, RecursionError) as exc: raise ValueError(f"Invalid execution state: {exc}") from exc @@ -273,18 +331,31 @@ def active_step(tree): return None -def scope_summaries(tree): - """Report workflow boundaries without exposing private inputs or results.""" +def scope_summaries(tree, run_status="running"): + """Report workflow boundaries without exposing private inputs or results. + + Only an unfinished scope containing the active occurrence inherits a + terminal run status; completed scopes retain their recorded result. + """ summaries = [] + active = active_step(tree) + active_path = active[0] if active is not None else () for _, node, path, _ in walk_execution(tree["sequence"]): binding = node.get("binding") if binding: output = node.get("result", {}).get("output", {}) + status = output.get("status") + if status is None: + status = ( + run_status + if run_status in HALTING and active_path[:len(path)] == path + else "running" + ) summaries.append( { "scope_path": path, "workflow_id": binding["workflow"], - "status": output.get("status", "running"), + "status": status, } ) return summaries @@ -317,6 +388,39 @@ def loop_alias_for(kind, qualified, iteration): return (qualified, iteration) if kind in LOOP_TYPES and iteration else None +def result_view(node): + """Public result, distinct from the frozen result used to execute children.""" + result = node["result"] + if "fan_results" in node: + return {**result, "output": {**result["output"], "results": node["fan_results"]}} + return result + + +@dataclass(frozen=True) +class Occurrence: + """One invocation, never stored on a shared step implementation.""" + + config: dict + node: dict + context: StepContext + ancestry: tuple + path: tuple + public: bool + qualified: str + + +@dataclass(frozen=True) +class SubtreeResult: + outcome: str + error: str | None + outputs: list | None = None + + +@dataclass(frozen=True) +class MissingImplementation: + error: str + + class Execution: def __init__(self, engine, state, registry, *, rebind=False): self.engine, self.state, self.registry = engine, state, registry @@ -335,31 +439,91 @@ def project_alias(self, context, name, result): """Project a fan-out item result into its enclosing workflow scope.""" context.steps[name] = result - def commit( + def transition( self, - node=None, + operation, + occurrence, changes=None, *, - context=None, - name=None, - public=False, - qualified=None, + publish=False, + announce=True, ): - """Mutate an occurrence and its compatibility views in one checkpoint.""" + """The only writer of occurrence progress and its checkpointed views. + + Phase is the continuation point; active marks entry into that continuation. + Own-step results and subtree outcomes deliberately remain distinct. + Validate the candidate before changing memory or writing anything. + """ + node = occurrence.node with self.state._lock: if self.state._checkpoint_failed: raise CheckpointError("A previous checkpoint failed") - if node is not None: - node.update(changes or {}) - if context is not None and "result" in node: - self.project( - context, - name, - node["result"], - public=public, - qualified=qualified or name, - ) - self.state.save() + phases, fields = TRANSITIONS[operation] + if node["phase"] not in phases: + raise ValueError(f"Invalid {operation} transition from {node['phase']}") + if set(changes or {}) - fields: + raise ValueError(f"Invalid fields for {operation} transition") + candidate = {**node, **(changes or {})} + if operation in {"expand", "bind"}: + candidate["phase"] = "children" + elif operation == "outputs": + candidate["phase"] = "outputs" + elif operation in {"finish", "settle"}: + candidate["phase"] = ( + "done" if candidate["outcome"] == "completed" + else "blocked" if operation == "finish" else "children" + ) + if operation == "begin": + candidate["active"] = True + elif operation in {"finish", "settle", "leave"}: + candidate["active"] = False + previous = node.get("children", []) + new_children = () + if "children" in (changes or {}): + existing = {id(child) for child in previous} + new_children = tuple( + child for child in candidate["children"] if id(child) not in existing + ) + # The same node rules are checked when a checkpoint is loaded. + check_node(occurrence.config, candidate, changed=set(changes or ()), + new_children=new_children) + node.update(candidate) + if operation == "begin": + self.state.current_step_id = occurrence.qualified + if publish and "result" in node: + self.project( + occurrence.context, + occurrence.config["id"], + result_view(node), + public=occurrence.public, + qualified=occurrence.qualified, + ) + # Leave is memory-only and best-effort; the run-level handler + # checkpoints the unwound tree on an exception. + # Do not insert a second checkpoint between completion and its event. + if operation != "leave": + self.state.save() + if announce: + self.notify(operation, occurrence, publish=publish) + + def notify(self, operation, occurrence, *, publish): + """Post-checkpoint notifications; never responsible for persistence.""" + config, node = occurrence.config, occurrence.node + kind = config.get("type", "command") + args = occurrence.qualified, occurrence.path, occurrence.ancestry + if operation == "begin": + label = kind if kind == "workflow" else config.get("command", "") or kind + self.emit("step_started", *args, type=kind, callback_label=label) + elif operation in {"expand", "finish"}: + result = node["result"] + if publish: + self.emit("step_completed", *args, status=result["status"]) + if result["status"] == "failed": + event = { + "aborted": "workflow_aborted", + "completed": "step_continue_on_error", + }.get(node.get("outcome"), "step_failed") + self.emit(event, *args, error=result.get("error")) def emit( self, @@ -380,24 +544,6 @@ def emit( with self.engine._callback_lock: self.engine.on_step_start(qualified, callback_label) - def start(self, config, qualified, path, ancestry): - """Persist an occurrence as the active step, then announce its start. - - Every step path starts here, so the checkpoint always precedes the - ``step_started`` event and callback, as on main: status reports the - occurrence while it runs and after a crash, and a failed checkpoint - leaves no start event behind. - """ - kind = config.get("type", "command") - with self.state._lock: - self.state.current_step_id = qualified - self.commit() - # A call is labelled by its type; a stray ``command`` key must not rename it. - label = kind if kind == "workflow" else config.get("command", "") or kind - self.emit( - "step_started", qualified, path, ancestry, type=kind, callback_label=label - ) - def run( self, seq, @@ -421,7 +567,6 @@ def run( self.state.current_step_index = index + self.state.execution.get( "offset", 0 ) - self.state.current_step_id = name outcome = self.step( config, node, @@ -457,7 +602,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): kind = config.get("type", "command") if node["phase"] == "done": self.project( - context, name, node["result"], public=public, qualified=qualified + context, name, result_view(node), public=public, qualified=qualified ) if kind == "fan-out" and node.get("children"): self.fan_out( @@ -480,55 +625,53 @@ def step(self, config, node, context, ancestry, path, public, qualified): # its fan-out item keeps its published output (see run_item). if node.get("result") is not None: self.project( - context, name, node["result"], public=public, qualified=qualified + context, name, result_view(node), public=public, qualified=qualified ) return "aborted" - if kind == "workflow": - return self.workflow( - config, - node, - context, - ancestry, - path, - public, - qualified, + occurrence = Occurrence(config, node, context, ancestry, path, public, qualified) + calls = kind == "workflow" + try: + # Workflow calls are announced on every entry, including resumed bound calls. + self.transition( + "begin", occurrence, + announce=node["phase"] in {"ready", "blocked"} or calls, ) + result = self.workflow(occurrence) if calls else self.execute_step(occurrence) + return self.finish(occurrence, result) + except BaseException: + if node.get("active"): + try: + self.transition("leave", occurrence) + except Exception: + pass # A crash-shaped active node is valid on resume. + raise + + def execute_step(self, occurrence): + config, node, context = occurrence.config, occurrence.node, occurrence.context + ancestry, path = occurrence.ancestry, occurrence.path + public, qualified = occurrence.public, occurrence.qualified + name, kind = config["id"], config.get("type", "command") if node["phase"] in {"ready", "blocked"}: - self.start(config, qualified, path, ancestry) impl = self.registry.get(kind) if impl is None: # As on main: terminal, only step_failed, no projected result. # The node keeps its result so resume can retry after reinstalling. error = unknown_step_error(kind) - result = StepResult(StepStatus.FAILED, error=error) - self.commit( - node, - { - "phase": "blocked", - "result": self.record(config, result, context), - "outcome": "failed", - "error": error, - }, - ) - self.emit("step_failed", qualified, path, ancestry, error=error) - return "failed" + return MissingImplementation(error) result = impl.execute(config, context) + expansion = {} + if kind == "fan-out": + # Definitions belong to the tree, never to JSON result records. + output = dict(result.output) + template = output.pop("step_template", {}) + expansion["template"] = yaml.safe_dump(template, sort_keys=False) + result = replace(result, output=output) if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: - return self.finish( - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ) + return result children = [sequence(result.next_steps)] if result.next_steps else [] if kind == "fan-out": - template = result.output.get("step_template", {}) children = ( [occurrences([template]) for _ in result.output.get("items", [])] if template @@ -538,44 +681,21 @@ def step(self, config, node, context, ancestry, path, public, qualified): if not children: if kind == "fan-out": result.output = {**result.output, "results": []} - return self.finish( - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ) - self.commit( - node, - {"phase": "children", "result": data, "children": children}, - context=context, - name=name, - public=public, - qualified=qualified, - ) - self.emit( - "step_completed", - qualified, - path, - ancestry, - status=result.status.value, + return result + self.transition( + "expand", occurrence, + {"result": data, "children": children, **expansion}, + publish=True, ) else: self.project( - context, name, node["result"], public=public, qualified=qualified + context, name, result_view(node), public=public, qualified=qualified ) if kind == "fan-out": outcome, error, outputs = self.fan_out( config, node, context, ancestry, path, public, qualified ) - data = { - **node["result"], - "output": {**node["result"]["output"], "results": outputs}, - } else: outcome, error = self.run_children( config, node, context, ancestry, path, public, qualified @@ -588,7 +708,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): config.get("condition", False), context ): child = occurrences(steps_of(node["children"][0])) - self.commit(node, {"children": [*node["children"], child]}) + self.transition("iterate", occurrence, {"children": [*node["children"], child]}) outcome, error = self.run( child, context, @@ -602,20 +722,7 @@ def step(self, config, node, context, ancestry, path, public, qualified): ) if outcome in HALTING: break - self.commit( - node, - { - "phase": "done" if outcome == "completed" else "children", - "outcome": outcome, - "error": error, - **({"result": data} if kind == "fan-out" else {}), - }, - context=context if kind == "fan-out" else None, - name=name if kind == "fan-out" else None, - public=public, - qualified=qualified, - ) - return outcome + return SubtreeResult(outcome, error, outputs if kind == "fan-out" else None) @staticmethod def record(config, result, context): @@ -644,74 +751,42 @@ def record(config, result, context): ) return data - def finish( - self, - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ): - name = config.get("id", "step-0") + def finish(self, occurrence, result): + config, context = occurrence.config, occurrence.context + if isinstance(result, SubtreeResult): + self.transition( + "settle", occurrence, + {"outcome": result.outcome, "error": result.error, + **({"fan_results": result.outputs} if result.outputs is not None else {})}, + publish=result.outputs is not None, + ) + return result.outcome + missing = isinstance(result, MissingImplementation) + if missing: + result = StepResult(StepStatus.FAILED, error=result.error) outcome = "completed" if result.status == StepStatus.PAUSED: outcome = "paused" elif result.status == StepStatus.FAILED: outcome = "aborted" if result.output.get("aborted") else "failed" - if outcome == "failed" and config.get("continue_on_error") is True: + if not missing and outcome == "failed" and config.get("continue_on_error") is True: outcome = "completed" - self.commit( - node, + self.transition( + "finish", occurrence, { - "phase": "done" if outcome == "completed" else "blocked", "result": self.record(config, result, context), "outcome": outcome, "error": result.error, }, - context=context, - name=name, - public=public, - qualified=qualified, - ) - self.emit( - "step_completed", - qualified, - path, - ancestry, - status=result.status.value, + publish=not missing, ) - if result.status == StepStatus.FAILED: - event = { - "aborted": "workflow_aborted", - "completed": "step_continue_on_error", - }.get(outcome, "step_failed") - self.emit( - event, - qualified, - path, - ancestry, - error=result.error, - ) return outcome - def workflow( - self, - config, - node, - context, - ancestry, - path, - public, - qualified, - ): + def workflow(self, occurrence): from .engine import WorkflowDefinition, workflow_dir_for - # Unlike other steps, a call is announced on every entry, including a - # resumed call that is already bound and skips the binding checkpoint. - self.start(config, qualified, path, ancestry) + config, node, context = occurrence.config, occurrence.node, occurrence.context + ancestry, path = occurrence.ancestry, occurrence.path binding = node.get("binding") target = binding["workflow"] if binding else config.get("workflow") if binding is None: @@ -725,31 +800,20 @@ def workflow( "inputs": bind_inputs(self.engine, definition, config, context), "workflow_dir": workflow_dir_for(definition), } - self.commit( - node, + self.transition( + "bind", occurrence, { "binding": binding, "children": [occurrences(definition.steps)], - "phase": "children", }, ) except CallError as exc: target = target if isinstance(target, str) else repr(target) - result = StepResult( + return StepResult( StepStatus.FAILED, output={"workflow": target, "status": "failed", "error": str(exc)}, error=str(exc), ) - return self.finish( - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ) else: definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) if self.rebind: @@ -757,7 +821,7 @@ def workflow( self.engine, definition, config, context, binding["inputs"] ) binding = {**binding, "inputs": inputs} - self.commit(node, {"binding": binding}) + self.transition("rebind", occurrence, {"binding": binding}) child_context = child_context_for_call(context, binding, definition) outcome, error = self.run( node["children"][0], @@ -769,22 +833,12 @@ def workflow( ) output = {"workflow": target, "status": outcome} if outcome == "completed": - self.commit(node, {"phase": "outputs"}) + self.transition("outputs", occurrence) try: output.update(evaluate_outputs(definition, child_context)) except CallError as exc: output.update(status="failed", error=str(exc)) - result = StepResult(StepStatus.FAILED, output=output, error=str(exc)) - return self.finish( - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ) + return StepResult(StepStatus.FAILED, output=output, error=str(exc)) elif outcome == "aborted": output["aborted"] = True if error is not None: @@ -796,17 +850,7 @@ def workflow( if outcome == "paused" else StepStatus.FAILED ) - result = StepResult(status, output=output, error=error) - return self.finish( - config, - node, - result, - context, - ancestry, - path, - public, - qualified, - ) + return StepResult(status, output=output, error=error) def fan_out( self, config, node, context, ancestry, path, public, qualified, *, sequential=False @@ -821,14 +865,11 @@ def fan_out( workers = 1 workers = min(workers, len(items)) initial = deepcopy(context.steps) - # Items run before the engine adds ``results`` to the fan-out record. A - # resumed fan-out's checkpointed record carries the partial results, so - # hide them from item contexts; the reporting view keeps them. + # Children always consume the frozen expansion result. Aggregate outputs + # belong to the reporting view, never to the item execution context. for key in {config.get("id", "step-0"), qualified}: - record = initial.get(key) - if isinstance(record, dict) and "results" in record.get("output", {}): - output = {k: v for k, v in record["output"].items() if k != "results"} - initial[key] = {**record, "output": output} + if key in initial: + initial[key] = deepcopy(node["result"]) halted = threading.Event() template_name = child_steps(config, node, 0)[0]["id"] @@ -851,6 +892,8 @@ def run_item(index): if outcome in HALTING: halted.set() record = child["nodes"][0].get("result") + if record is not None: + record = result_view(child["nodes"][0]) # Expose only results projected by the item traversal. Missing step # implementations keep an internal retry record but publish nothing. if local.steps.get(local_name) is inherited: diff --git a/src/specify_cli/workflows/command_status.py b/src/specify_cli/workflows/command_status.py index f58c990698..97471eb10c 100644 --- a/src/specify_cli/workflows/command_status.py +++ b/src/specify_cli/workflows/command_status.py @@ -93,7 +93,7 @@ def workflow_status( if state.execution: from ._execution import scope_summaries - for scope in scope_summaries(state.execution): + for scope in scope_summaries(state.execution, state.status.value): path = " / ".join(scope["scope_path"]) cli.console.print( f" {cli._escape_markup(path)} → " diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index 384fdbbb79..50be13033e 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -1174,12 +1174,13 @@ def resume( raise ValueError(msg) if state.execution is not None: - from ._execution import steps_of + from ._execution import validate_execution - persisted_steps = steps_of(state.execution["sequence"]) - offset = state.execution.get("offset", 0) - if persisted_steps != definition.steps[offset:]: - raise ValueError("Invalid execution state: root sequence differs from workflow snapshot") + validate_execution( + state.execution, + workflow_steps=definition.steps, + current_step_index=state.current_step_index, + ) dispatch_default_errors = _dispatch_default_errors(definition) if dispatch_default_errors: diff --git a/tests/specify_cli/workflows/test_command_status.py b/tests/specify_cli/workflows/test_command_status.py index 16f6a01352..73994d5b22 100644 --- a/tests/specify_cli/workflows/test_command_status.py +++ b/tests/specify_cli/workflows/test_command_status.py @@ -127,6 +127,55 @@ def test_composed_gate_status_and_resume(self, project_dir): assert resumed.exit_code == 0 assert json.loads(resumed.stdout)["status"] == "completed" + @pytest.mark.parametrize("failure, expected", [ + (KeyboardInterrupt, "paused"), + (RuntimeError, "failed"), + ]) + def test_status_reports_interrupted_call_in_json_and_text( + self, project_dir, monkeypatch, failure, expected + ): + from specify_cli.workflows import STEP_REGISTRY + from specify_cli.workflows.base import StepBase + from specify_cli.workflows.catalog import WorkflowRegistry + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + child_dir = project_dir / ".specify" / "workflows" / "child" + child_dir.mkdir(parents=True) + (child_dir / "workflow.yml").write_text( + "workflow: {id: child, name: Child}\n" + "steps:\n - {id: work, type: explode}\n", + encoding="utf-8", + ) + WorkflowRegistry(project_dir).add("child", {"enabled": True}) + engine = WorkflowEngine(project_dir) + root = WorkflowDefinition({ + "workflow": {"id": "parent", "name": "Parent"}, + "steps": [{"id": "call", "type": "workflow", "workflow": "child"}], + }) + if failure is RuntimeError: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="interrupted-call") + else: + engine.execute(root, run_id="interrupted-call") + + status = self._invoke(project_dir, [ + "workflow", "status", "interrupted-call", "--json", + ]) + assert status.exit_code == 0, status.output + assert json.loads(status.stdout)["workflow_scopes"] == [ + {"scope_path": ["call"], "workflow_id": "child", "status": expected} + ] + human = self._invoke(project_dir, ["workflow", "status", "interrupted-call"]) + assert human.exit_code == 0, human.output + assert f"child: {expected}" in human.stdout + class TestWorkflowCliAlignment: diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py index c8f0250450..9c6f669212 100644 --- a/tests/workflows/test_composition_execution.py +++ b/tests/workflows/test_composition_execution.py @@ -461,6 +461,22 @@ def test_legacy_resume_adapts_once(tmp_path, probe): assert probe == {"before": 1, "wait": 3} +def test_version_one_tree_rejected_without_writes(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["version"] = 1 + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + + with pytest.raises(ValueError, match="Unsupported execution version"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + @pytest.mark.parametrize( "mutation", [ @@ -761,6 +777,38 @@ def test_interrupt_during_resume_checkpoint_pauses_run(tmp_path, monkeypatch, pr assert probe == {"wait": 2, "next": 1} +@pytest.mark.parametrize("failure", [KeyboardInterrupt, RuntimeError]) +def test_leave_failure_preserves_step_exception(tmp_path, monkeypatch, failure): + from specify_cli.workflows._execution import Execution + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + original_transition = Execution.transition + + def transition(self, operation, occurrence, *args, **kwargs): + if operation == "leave": + raise ValueError("leave failed") + return original_transition(self, operation, occurrence, *args, **kwargs) + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + monkeypatch.setattr(Execution, "transition", transition) + engine = WorkflowEngine(tmp_path) + root = definition("parent", [{"id": "work", "type": "explode"}]) + if failure is KeyboardInterrupt: + state = engine.execute(root, run_id="leave-failure") + assert state.status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + else: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="leave-failure") + state = RunState.load("leave-failure", tmp_path) + assert state.error == "boom" + + def test_checkpoint_failure_stops_concurrent_fan_out_logging(tmp_path, monkeypatch): import specify_cli.workflows._execution as execution from specify_cli.workflows._execution import CheckpointError @@ -778,22 +826,26 @@ def execute(self, config, context): assert first_committed.wait(5) return StepResult(output={"value": context.item}) - original_commit = execution.Execution.commit + original_transition = execution.Execution.transition + original_notify = execution.Execution.notify original_write = RunState._atomic_write_json - def commit(self, *args, **kwargs): + def transition(self, *args, **kwargs): item = items.get(threading.current_thread()) try: - original_commit(self, *args, **kwargs) + original_transition(self, *args, **kwargs) except CheckpointError: if item == 2: second_failed.set() raise - if item == 1 and not first_committed.is_set(): + + def notify(self, operation, occurrence, **kwargs): + if items.get(threading.current_thread()) == 1 and operation == "finish": # Item 1 committed and released the lock; item 2 now fails its # checkpoint before item 1 emits its completion event. first_committed.set() assert second_failed.wait(5) + return original_notify(self, operation, occurrence, **kwargs) def write(path, data): if items.get(threading.current_thread()) == 2: @@ -801,7 +853,8 @@ def write(path, data): original_write(path, data) monkeypatch.setitem(STEP_REGISTRY, "sync", Sync()) - monkeypatch.setattr(execution.Execution, "commit", commit) + monkeypatch.setattr(execution.Execution, "transition", transition) + monkeypatch.setattr(execution.Execution, "notify", notify) monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) with pytest.raises(CheckpointError): WorkflowEngine(tmp_path).execute( @@ -1226,15 +1279,15 @@ def test_replay_does_not_commit_completed_nodes(tmp_path, monkeypatch, probe): ) ) assert state.status == RunStatus.PAUSED - original = execution.Execution.commit + original = execution.Execution.transition committed = [] - def commit(self, node=None, changes=None, **kwargs): - if node is not None and node["phase"] == "done": - committed.append(node) - return original(self, node, changes, **kwargs) + def transition(self, operation, occurrence, changes=None, **kwargs): + if occurrence.node["phase"] == "done": + committed.append(occurrence.node) + return original(self, operation, occurrence, changes, **kwargs) - monkeypatch.setattr(execution.Execution, "commit", commit) + monkeypatch.setattr(execution.Execution, "transition", transition) state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) @@ -1996,6 +2049,39 @@ def test_nested_gate_reporting_uses_active_occurrence(tmp_path, probe): assert payload["workflow_scopes"][0]["status"] == "paused" +@pytest.mark.parametrize("failure, expected", [ + (KeyboardInterrupt, "paused"), + (RuntimeError, "failed"), +]) +def test_interrupted_bound_call_reports_run_outcome(tmp_path, monkeypatch, probe, failure, expected): + from specify_cli.workflows._commands import _workflow_run_payload + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + install(tmp_path, definition("first", [{"id": "done", "type": "probe"}])) + install(tmp_path, definition("child", [{"id": "work", "type": "explode"}])) + engine = WorkflowEngine(tmp_path) + root = definition("parent", [call("first", id="prior"), call()]) + if failure is RuntimeError: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="interrupted-call") + else: + assert engine.execute(root, run_id="interrupted-call").status == RunStatus.PAUSED + + saved = RunState.load("interrupted-call", tmp_path) + payload = _workflow_run_payload(saved) + assert payload["status"] == expected + assert payload["workflow_scopes"] == [ + {"scope_path": ["prior"], "workflow_id": "first", "status": "completed"}, + {"scope_path": ["call"], "workflow_id": "child", "status": expected}, + ] + + def test_rebind_failure_has_one_failed_caller_outcome(tmp_path, probe): install( tmp_path, @@ -2304,6 +2390,162 @@ def test_native_yaml_definition_and_long_id_roundtrip(tmp_path, probe): assert WorkflowEngine(tmp_path).resume(state.run_id).status == RunStatus.PAUSED +@pytest.mark.parametrize("scope", ["root", "if", "workflow", "fan-out"]) +def test_yaml_native_gate_template_survives_checkpoint_and_resume(tmp_path, monkeypatch, scope): + monkeypatch.setattr("sys.stdin.isatty", lambda: False) + gate = {"id": "wait", "type": "gate", "message": date(2026, 1, 1)} + steps = [gate] + if scope == "if": + steps = [{"id": "branch", "type": "if", "condition": True, "then": steps}] + elif scope == "workflow": + install(tmp_path, definition("child", steps)) + steps = [call()] + elif scope == "fan-out": + steps = [{"id": "fan", "type": "fan-out", "items": [0, 1], "step": gate}] + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", steps)) + assert state.status == RunStatus.PAUSED + state = engine.resume(state.run_id) + assert state.status == RunStatus.PAUSED + from specify_cli.workflows._execution import walk_execution + + gates = [config for config, _, _, _ in walk_execution(state.execution["sequence"]) + if config.get("type") == "gate"] + assert gates and all(config["message"] == date(2026, 1, 1) for config in gates) + + +@pytest.mark.parametrize("offset", [1, 2, 3]) +def test_resume_rejects_offset_skipping_blocked_step_without_writes(tmp_path, probe, offset): + steps = [{"id": "wait", "type": "probe", "await": True}, + {"id": "later", "type": "probe"}] + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", steps)) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"].update(offset=offset, sequence={ + "source": yaml.safe_dump(steps[offset:]), + "nodes": [{"phase": "ready"} for _ in steps[offset:]], + }) + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError, match="offset"): + engine.resume(state.run_id) + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + assert probe == {"wait": 1} + + +@pytest.mark.parametrize("scope", ["step", "if", "workflow", "fan-out"]) +def test_lifecycle_start_is_tree_backed_and_replay_does_not_restart(tmp_path, probe, scope): + from specify_cli.workflows._execution import walk_execution + + work = {"id": "work", "type": "probe"} + steps = [work] + if scope == "if": + steps = [{"id": "branch", "type": "if", "condition": True, "then": steps}] + elif scope == "workflow": + install(tmp_path, definition("child", steps)) + steps = [call()] + elif scope == "fan-out": + steps = [{"id": "fan", "type": "fan-out", "items": [0, 1], + "max_concurrency": 2, "step": work}] + steps.append({"id": "wait", "type": "probe", "await": True}) + engine = WorkflowEngine(tmp_path) + announced = [] + + def started(step_id, label): + saved = RunState.load("lifecycle", tmp_path) + matches = [node for _, node, _, name in walk_execution(saved.execution["sequence"]) + if name == step_id] + assert any(node.get("active") for node in matches) + announced.append(step_id) + + engine.on_step_start = started + state = engine.execute(definition("parent", steps), run_id="lifecycle") + assert state.status == RunStatus.PAUSED + assert all(not node.get("active") for _, node, _, _ in walk_execution(state.execution["sequence"])) + before = probe.copy() + announced.clear() + engine.resume(state.run_id) + assert announced == ["wait"] + assert probe["work"] == before["work"] + + +@pytest.mark.parametrize("mutation", [ + lambda node: node.update(active=True), + lambda node: node.update(active="yes"), + lambda node: node.update(template="- invalid\n"), + lambda node: node.update(fan_results={}), + lambda node: node.update(fan_results={"a": 1}), +]) +def test_lifecycle_rejects_invalid_completed_fan_out_without_writes(tmp_path, probe, mutation): + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", [ + {"id": "fan", "type": "fan-out", "items": [0], + "step": {"id": "work", "type": "probe"}}, + {"id": "wait", "type": "probe", "await": True}, + ])) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]["sequence"]["nodes"][0]) + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError): + engine.resume(state.run_id) + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + +@pytest.mark.parametrize("field,value", [("fan_results", []), ("template", "{}\n")]) +def test_non_fan_out_rejects_fan_out_fields(tmp_path, probe, field, value): + state = WorkflowEngine(tmp_path).execute(definition("parent", [ + {"id": "work", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ])) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][0][field] = value + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + + with pytest.raises(ValueError, match="Invalid fan-out"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + +@pytest.mark.parametrize("invalid", ["restart-completed", "override-phase", "finish-unfinished", "foreign-field"]) +def test_lifecycle_rejects_invalid_transitions_before_mutation(tmp_path, probe, invalid): + from copy import deepcopy + from specify_cli.workflows._execution import Execution, Occurrence + from specify_cli.workflows.base import StepContext + + config = {"id": "branch", "type": "if", "condition": True, + "then": [{"id": "wait", "type": "probe", "await": True}]} + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", [config])) + node = state.execution["sequence"]["nodes"][0] + occurrence = Occurrence(config, node, StepContext(), ("parent",), (0,), True, "branch") + executor = Execution(engine, state, STEP_REGISTRY) + if invalid == "restart-completed": + occurrence = Occurrence( + {"id": "done", "type": "probe"}, + {"phase": "done", "result": {"status": "completed", "output": {}}}, + StepContext(), ("parent",), (0,), True, "done", + ) + operation, changes = "begin", {} + elif invalid == "override-phase": + operation, changes = "begin", {"phase": "done"} + elif invalid == "foreign-field": + operation, changes = "iterate", {"result": {"status": "completed", "output": {}}} + else: + operation, changes = "settle", {"outcome": "completed", "error": None} + before_node = deepcopy(occurrence.node) + before_files = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError): + executor.transition(operation, occurrence, changes) + assert occurrence.node == before_node + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before_files + + def test_execution_shares_workflow_fan_out_and_loop_sources(tmp_path, probe): child = definition( "child", @@ -2492,6 +2734,32 @@ def size(iterations, length): assert large_many - small_many <= large_one - small_one + 256 +def test_loop_transition_validation_does_not_rewalk_prior_iterations( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + original = execution.yaml.safe_load + counts = [] + + def parse(source): + counts[-1] += 1 + return original(source) + + monkeypatch.setattr(execution.yaml, "safe_load", parse) + for iterations in (20, 40): + counts.append(0) + state = WorkflowEngine(tmp_path / str(iterations)).execute( + definition("parent", [{ + "id": "loop", "type": "do-while", "condition": True, + "max_iterations": iterations, + "steps": [{"id": "body", "type": "probe"}], + }]) + ) + assert state.status == RunStatus.COMPLETED + assert counts[1] < counts[0] * 2.5 + + def test_tree_backed_resume_has_no_setup_checkpoint(tmp_path, monkeypatch, probe): state = WorkflowEngine(tmp_path).execute( definition( diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index 9f559ce981..291c924467 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -89,6 +89,59 @@ transitions share one atomic state checkpoint; the inputs file is a compatibility mirror. A checkpoint failure prevents further writes by that executor instance. +### Occurrence lifecycle + +`Execution.step()` is the common runner for registered steps and workflow calls. +`execute_step()` and `workflow()` return a `StepResult`, a subtree outcome, or an +unknown-implementation failure. The runner alone performs `begin`, `finish`, +`settle`, and best-effort `leave` on exception unwinding; the phase and field +allow-lists in `transition()` reject invalid operations. The public, stateless +`StepBase.execute()` extension contract is unchanged. + +All occurrence mutations pass through `Execution.transition()`. It checks the +allowed source phase and fields, derives the destination phase, validates the +candidate with the same node rules used on load, projects results, and saves +under the run lock. Callers cannot supply a destination phase. + +| Operation | Meaning | +|-----------|---------| +| `begin` | Mark this occurrence active and checkpoint before `step_started` or its callback; a failed checkpoint leaves no start event | +| `expand` / `bind` | Freeze children and their source before child execution | +| `rebind` / `iterate` | Persist updated binding or the next loop occurrence | +| `outputs` | Children finished; declared workflow outputs remain to finalize | +| `finish` / `settle` | Record an own-step result or subtree outcome; clear activity | +| `leave` | Clear activity on exception unwinding; the run handler saves the failure/pause | + +`phase` identifies the continuation point, `active` identifies entered occurrences +(several may be active in a fan-out), and `outcome` identifies a subtree halt or +completion. A container's own result may be completed while its children are +paused. `current_step_id` is a compatibility status view, updated on entry and +reconciled against the tree on exit, rather than a resume cursor. +Status reporting uses a bound call's recorded result when available. If an +interruption or exception leaves the active call unfinished without a result, +its scope inherits the run's paused or failed status; completed calls keep +their own recorded status. + +`notify()` emits events and callbacks only after the corresponding checkpoint. +Persistence is mandatory lifecycle behavior, not a listener. Existing container +events describe completion of their own expansion; calls finish after their +children and declared outputs. Completed replay emits neither events nor saves. +Entering an unfinished container on resume checkpoints activity without repeating +its expansion event. + +Execution schema version 2 stores fan-out templates as shared YAML sources on +their parent occurrence. Raw `step_template` configuration is not published in +persisted step outputs. Frozen expansion results and aggregated `fan_results` +are separate: `result_view()` adds the aggregate for reporting and downstream +steps, while items always receive the frozen expansion view. This also preserves +YAML-native template scalars without putting them in JSON result records. + +Resume validates tree structure, root snapshot, and the legacy offset together, +before any writes. The offset must be within the workflow and no later than the +saved root index. Main-format checkpoints without a tree still adapt once; +private, unreleased version-1 trees are rejected rather than silently interpreted +as the new format. + ## Step Types The engine ships with 13 built-in step types, each in its own subpackage under `src/specify_cli/workflows/step/`: