Skip to content

feat(agno): capture resolved workflow HITL step requirements as user input messages (#906) - #1009

Open
VimalN2005 wants to merge 4 commits into
open-telemetry:mainfrom
VimalN2005:feat/agno-workflow-hitl-step-requirements-906
Open

VimalN2005 wants to merge 4 commits into
open-telemetry:mainfrom
VimalN2005:feat/agno-workflow-hitl-step-requirements-906

Conversation

@VimalN2005

Copy link
Copy Markdown
Contributor

Problem

In #731, Workflow.continue_run and Workflow.acontinue_run were instrumented to emit invoke_workflow spans. However, resolved human-in-the-loop (HITL) step requirements (StepRequirement) were omitted from gen_ai.input.messages because workflow steps are orchestration checkpoints rather than LLM tool calls (role: "tool"), as described in #906.

Solution

  • Map resolved StepRequirement objects during Workflow.continue_run and Workflow.acontinue_run to gen_ai.input.messages with role: "user", representing human interaction and input that resumes the workflow.
  • Handled attributes on resolved requirements:
    • step_id & step_name
    • confirmed, confirmation_message, rejection_feedback
    • user_input
    • selected_choices
    • output_reviewed and edited_output
  • Respect content capture toggle: when content capture is disabled (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false), step requirements are omitted from spans.
  • Follow Agno convention: when reading from run_response.step_requirements, only the active (latest) step requirement is captured.
  • Added comprehensive unit tests and towncrier newsfragment.

Fixes #906

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Tests

…input messages (open-telemetry#906)

In Agno workflows, human-in-the-loop (HITL) step reviews are represented
by StepRequirement objects passed via step_requirements or run_response.
Since workflow steps are orchestration checkpoints rather than LLM tool
calls, resolved step requirements were previously omitted.

Capture resolved StepRequirement objects on Workflow.continue_run and
Workflow.acontinue_run spans as role: "user" input messages when content
capture is enabled, representing human decisions and inputs during workflow
orchestration.

Fixes open-telemetry#906

Signed-off-by: VimalN2005 <vimalsahani2005@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 14:53
@VimalN2005
VimalN2005 requested a review from a team as a code owner October 29, 2026 14:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Waiting on the author · refreshed 2026-11-01 22:38 UTC

Respond to 1 review item (e.g. link a commit, explain why not, ask a follow-up):

  • Top-level threads: 1
Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
  • Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

…ment (open-telemetry#906)

Address reviewer feedback:
- Retain only the last (active) requirement when step_requirements is
  passed explicitly or read from run_response, preventing duplicate
  history from earlier pause cycles.
- Extract executor-level HITL resolution details (tool_call_id, tool_name,
  confirmed, confirmation_note, result, user_input, user_feedback) from
  executor_requirements.
- Format code and tests with ruff to fix CI pre-commit failure.
- Add unit test for executor-level HITL requirements.

Signed-off-by: VimalN2005 <vimalsahani2005@gmail.com>
@VimalN2005

Copy link
Copy Markdown
Contributor Author

Thanks @DylanRussell! I reviewed semconv PR #188 — that proposal introduces individual step execution spans (run_step operation with gen_ai.step.name and duration metrics), but explicitly deferred step-level events and input.messages reshaping to a follow-up.

Workflow.continue_run and Workflow.acontinue_run instrument the workflow invocation span itself (invoke_workflow). Representing the human interaction/decision that resumes the workflow as a role: "user" input message on invoke_workflow provides immediate observability into the human inputs driving the continued run, while complying strictly with existing GenAI semconv for invocation spans. Once PR #188 lands and individual step spans are added here, having continuation inputs on invoke_workflow remains complementary. That said, if you'd prefer to wait or adjust the representation, happy to follow your guidance!

@lmolkova lmolkova left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! I'm not sure gen_ai.input.messages is the right place for this as is. The user message currently carries a JSON dump of Agno state (step_id, step_name, executor ids, tool_call_id, confirmation_message), and most of that isn't what the human said. confirmation_message is the developer-defined prompt shown to the human, and executor-level results are tool responses that belong on the inner agent's span.

Two options:

  • narrow it to the human's actual answers (confirmed, user_input, selected_choices, edited_output, rejection_feedback) with no framework metadata, or
  • park it until open-telemetry/semantic-conventions-genai#188 settles how workflow steps and HITL should look.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[agno] Telemetry representation for workflow HITL step requirements

5 participants