feat: add RAGAS adapter for third-party eval metrics

[brandonaxu]

Summary

Adds RAGASAdapter to the third-party evaluation adapters, alongside the existing DeepEvalAdapter and AutoEvalsAdapter. Wraps any RAGAS metric as an AgentCore code-based Lambda evaluator with the same BaseAdapter pattern.

Design

Scores metrics through RAGAS's per-sample APIs (metric.single_turn_score() / metric.multi_turn_score() for legacy metrics, metric.score(**kwargs) for ragas.metrics.collections metrics) rather than the batch ragas.evaluate() pipeline. This means the adapter itself adds no dependency on datasets/pyarrow/pandas, which makes it compatible with slim ragas builds for size-constrained Lambda deployments.

Three-way dispatch:

• Legacy single-turn (Faithfulness, ExactMatch, ContextPrecision, ...): SingleTurnSample + single_turn_score()
• Legacy multi-turn (ToolCallAccuracy, TopicAdherenceScore, AgentGoalAccuracy): MultiTurnSample + multi_turn_score()
• Collections / decorator metrics (ragas.metrics.collections, @discrete_metric, @numeric_metric): metric.score(**fields) with signature-based field filtering

Features

• Validates metric.required_columns before scoring — returns MISSING_REQUIRED_FIELD with actionable guidance instead of letting ragas return silent 0.0 scores
• Parses ground truth and retrieval context embedded in the user message (\n\nReference Answer:\n and \n\nContext:\n separators) since ADOT trace formats have no dedicated fields for these
• Recovers ranked chunk lists from JSON-serialized embedded contexts, preserving chunk boundaries for rank-aware metrics (LLMContextPrecision)
• Handles threshold=None metrics (e.g. SemanticSimilarity) and provides an adapter-level threshold= override for collections metrics which carry none
• Discrete (string-valued) metrics produce categorical labels natively
• Preserves MetricResult.reason as the explanation
• ImportError at score time returns MISSING_DEPENDENCY (not a generic crash)
• Multi-turn messages are stripped of embedded markers so ground truth doesn't leak into the conversation being judged
• RAGASAdapter is the canonical class; RagasAdapter alias provided

Usage

from bedrock_agentcore.evaluation.custom_code_based_evaluators import custom_code_based_evaluator
from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.ragas import RAGASAdapter
from ragas.metrics import Faithfulness
from ragas.llms import LangchainLLMWrapper
from langchain_aws import ChatBedrockConverse

eval_llm = LangchainLLMWrapper(ChatBedrockConverse(
    model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0",
    region_name="us-east-1",
))
adapter = RAGASAdapter(metric=Faithfulness(), llm=eval_llm)

@custom_code_based_evaluator()
def handler(eval_input, context=None):
    return adapter(eval_input, context)

Known Limitations

• reference_contexts (ground-truth contexts for reference-comparison metrics) is not defaulted from retrieved contexts — supply it via custom_mapper when a genuine ground-truth source exists
• reference_tool_calls built from expected_trajectory.toolNames carry names without arguments; arg-accurate comparison needs a custom_mapper with full ToolCall objects
• reference_topics (TopicAdherenceScore) has no span source — requires custom_mapper
• Dual-mode metrics (AspectCritic, SimpleCriteriaScore) always score single-turn
• ragas <1.0 imports datasets when the ragas package is imported; the adapter is compatible with trimmed builds but does not solve that import itself

Dependency Notes

• Pins ragas>=0.4.3,<1.0.0 — the tested version range
• Pins langchain-community>=0.3.0,<0.4.2 — langchain-community 0.4.2 removed chat_models.vertexai, which ragas <1.0 imports unconditionally
• Adds ragas to the evaluation integ test extra-deps in CI
• uv.lock regenerated for the new extra; incidental bumps to langchain-community resolution only

Testing

• 48 unit tests covering all dispatch branches, embedded parsing (plain text + JSON chunk lists), validation, multi-turn messages, threshold override, error codes
• 8 integration tests with real deterministic metrics (legacy ExactMatch, collections ExactMatch, @discrete_metric, ToolCallAccuracy match/mismatch, adapter-imports-without-datasets)
• All pre-commit hooks pass; uv lock --check passes

[jariy17]

YOu have merge conflicts and small nit

[jariy17]

Why can't we use this helper function instead

[brandonaxu]

Good catch — switched to _make_agent_evaluator_input in the latest push. The separate helper was a leftover from earlier development when the shared one used a span format the mappers couldn't parse, but that's been fixed on main since. All ragas integ tests should now use the shared helper.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.85417% with 31 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@5d4ca0d). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...code_based_evaluators/third_party/ragas/adapter.py	83.68%	24 Missing and 7 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #618   +/-   ##
=======================================
  Coverage        ?   88.56%           
=======================================
  Files           ?      120           
  Lines           ?    10023           
  Branches        ?     1527           
=======================================
  Hits            ?     8877           
  Misses          ?      763           
  Partials        ?      383           

Flag	Coverage Δ
unittests	88.56% <83.85%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aidandaly24]

[P1] I think collections multi-turn metrics bypass the field conversion here. _score_multi_turn() builds the RAGAS messages and reference_tool_calls, but collections metrics go directly to _score_collections() with a string user_input and no reference tools. Real RAGAS 0.4.3 collections.ToolCallAccuracy returned MISSING_REQUIRED_FIELD on the live Strands session despite expectedTrajectory.toolNames; the same metric scored 1.0 when given the converted fields. Could we prepare those fields for collections signatures too and test the real collections class? The current test uses the deprecated legacy class with a custom mapper, so it does not exercise this path.