fix: DeepEvalAdapter to extract multi-turn conversations from service-normalized SESSION format

[ybdarrenwang]

Background: PR #568

PR #568 (feat: third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers) introduced the DeepEvalAdapter — a generic wrapper that takes any DeepEval metric and runs it inside an AgentCore Lambda evaluator. The adapter uses strands-evals mappers to auto-detect span formats, extract fields, and construct the appropriate DeepEval test case (LLMTestCase for single-turn, ConversationalTestCase for multi-turn).

The Bug

The adapter's multi-turn extraction path relies on _session_to_span_map_result() which builds the turns list from multiple AgentInvocationSpan objects — one per trace/invocation in the session. This works when the strands-evals mapper successfully parses spans into the Session → Trace → AgentInvocationSpan hierarchy (i.e., when each conversation turn arrives as a separate span in the CloudWatch split format).

However, the AgentCore evaluation service normalizes spans before invoking Lambda at SESSION level. It collapses all ADOT span documents sharing the same session.id into a single span with multiple span_events entries — each span_event.body contains one conversation turn's input.messages and output.messages.

In this service-normalized format:

1. The strands-evals mapper either fails to detect the format (wrong/missing scope) or maps it into a Session with only one AgentInvocationSpan (because there's only one physical span)
2. _session_to_span_map_result() produces turns with only 2 entries (1 user + 1 assistant from the single AgentInvocationSpan)
3. The guard turns if len(turns) > 2 else None filters this to None
4. _build_conversational_test_case() sees result.turns is empty and raises: "Multi-turn metric requires multiple conversation turns but only a single turn was found"

The fallback _extract_from_service_format() only handled single-turn extraction from gen_ai semantic convention events — it had no logic for parsing span_events[*].body.

Impact: Every multi-turn metric failed at SESSION level because no turns were extracted at all, forcing users to bypass the adapter and write custom conversational handlers.

The Fix

The fix extends _extract_from_service_format() in registry.py to handle the service-normalized SESSION format:

1. New helper _extract_message_text() — parses the nested service message structure ({content: {content: [{text: ...}]}} and the {content: {message: [{text: ...}]}} variant) into plain text.

2. Multi-turn extraction in _extract_from_service_format() — added a new code path that runs first:

   • Scans each span for span_events with ≥ 1 entry
   • Iterates each span_event.body, extracts user input and assistant output from body.input.messages / body.output.messages
   • Builds the turns list (with turns=None when only 1 turn pair exists, matching existing semantics)
   • Returns a SpanMapResult with populated turns field

3. Test expectation update — Two existing error-handling tests (test_15_unrecognized_scope_deepeval, test_02_unrecognized_scope in autoevals) changed from FIELD_EXTRACTION_ERROR to MISSING_REQUIRED_FIELD because the mapper now succeeds in parsing the span (no extraction error) but finds empty input/output fields (triggering the more specific "missing required field" error).

The fix is backward-compatible: the single-turn gen_ai events path remains unchanged as a fallback, and the multi-turn path only activates when span_events entries are present.

Scope

This PR restores turn extraction (role + content) from the service-normalized
SESSION format. That is sufficient for the multi-turn metrics that only need the
conversation turns:

• GoalAccuracy, RoleAdherence, ConversationCompleteness, KnowledgeRetention,
  TopicAdherence, TurnRelevancy, ConversationalGEval

Out of Scope / Known Limitation (follow-up)

Metrics that need per-turn side data are NOT fixed by this PR, because the
service-normalized span_events[*].body format has no slot for that data — only
input.messages and output.messages:

• retrieval_context → TurnContextualPrecision, TurnContextualRecall, TurnFaithfulness
• tools_called → ToolUse

_build_conversational_test_case() constructs Turn(role, content) only, so even
if the body carried these fields, they would be dropped. Fixing this requires a
format convention for where per-turn retrieval_context / tools_called live in
span_events[*].body (e.g. body.retrieval_context, body.tools), then threading
them through SpanMapResult.turns[i] into the Turn(...) constructor. Tracked as
a follow-up.

Testing

Added 9 new test cases:

test_span_mappers.py — TestServiceNormalizedMultiTurn (6 tests):

• Extracts all turns from multi-event span_events
• Input/output reflect the last turn
• Single span_event returns None turns (not multi-turn)
• Handles plain string content variant
• Handles nested {content: {content: [{text: ...}]}} variant
• Realistic 5-turn session evaluation

test_adapter.py — TestDeepEvalAdapterServiceNormalizedMultiTurn (3 tests):

• Conversational metric receives all turns with correct count
• Turn content is correctly extracted from nested message format
• End-to-end 5-turn session evaluation passes through adapter

Note: RAG-context / tool-use multi-turn metrics are intentionally not covered
(see Out of Scope) — they require a format change, tracked separately.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[stone-coding]

Reviewed PR. The fix correctly handles the SESSION-level service-normalized format where multiple turns are collapsed into span_events[*].body. My original _extract_from_service_format() only covered single-turn gen_ai events.

[aidandaly24]

[P0] I think this still misses the exact one-span/many-event case this PR is trying to fix. When the CloudWatch mapper recognizes the span, it returns valid input/output from the first event, so the fallback in map_spans() never calls this new path. I reproduced this using real Strands invoke_agent bodies collapsed into the target shape: turns stayed None and DeepEval returned FIELD_EXTRACTION_ERROR. Could we run the service-format extraction when multiple body events are present even if the mapper found one valid pair, and retain the real scope/attributes in the regression test?

[aidandaly24]

[P0] I do not think this expected error code matches the current behavior. The full third-party suite has 2 failed, 115 passed: this case and the equivalent DeepEval test both still return FIELD_EXTRACTION_ERROR. These spans have no body or span_events, so the new extractor cannot reach MISSING_REQUIRED_FIELD. The base tests pass with the original expectation. Could we keep FIELD_EXTRACTION_ERROR unless the runtime behavior is intentionally changing too?

[aidandaly24]

[P0] Could we reuse the existing CloudWatch message parsing here? Real Strands bodies encode user content as {"content": "[{\"text\": ...}]"}, and multi-turn input can contain interleaved user/assistant history. This helper returns the serialized JSON literally and selects the first historical message. Against three real invoke_agent events, it produced six turns but every user turn still contained JSON syntax. PR #454 also established the chronological-role/latest-user behavior, so sharing that parsing logic would keep this path consistent.