feat: Revision third-party eval metrics adapter (DeepEval + Autoevals) by stone-coding · Pull Request #568 · aws/bedrock-agentcore-sdk-python

stone-coding · 2026-07-06T17:07:35Z

Summary

Add DeepEvalAdapter and AutoevalsAdapter that integrate third-party evaluation metrics with
AgentCore's code-based evaluator framework
Built-in StrandsSpanMapper (class-based) auto-extracts input/output from Strands agent
OTel spans — supports both inline events (unified ADOT) and span body (CloudWatch ADOT) formats
customer_mapper parameter lets customers bypass the built-in mapper and return the metric
library's native type directly:
- DeepEval: customer_mapper: Callable[[EvaluatorInput], LLMTestCase]
- Autoevals: customer_mapper: Callable[[EvaluatorInput], Dict[str, Any]]
Simplified SpanMapResult — only fields metrics actually consume (input, actual_output,
retrieval_context, system_prompt, expected_output)
Removed separate validate_fields() step — validation happens inside each adapter's _run()
method when building the test case
_filter_spans_by_target applied only for built-in mapper path (not for customer mapper)
Never raises unhandled exceptions — all error paths return valid EvaluatorOutput with
structured errorCode/errorMessage

Test plan

83 unit tests passing (pytest tests/.../third_party/ -v)
E2E on-demand: built-in mapper → Score 1.0 Pass (Strands healthcare agent)
E2E on-demand: custom mapper → Score 1.0 Pass (non-Strands spans)
E2E online eval: pipeline invokes Lambda, score 0.98 Pass (CloudWatch results)
Error handling: 4/4 (malformed, unsupported scope, empty attrs, success schema)

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

…leTurnParams deprecation

…d EvaluatorInput support

… layer, simplify per TJ/Irene feedback

…ion, add validate_fields to DeepEvalAdapter

…xpected_response_text property

- Rename span_parsers → span_mappers, simplify to Strands-only - Rename field_mapper → customer_mapper across all adapters - customer_mapper now returns native types directly: - DeepEval: EvaluatorInput → LLMTestCase - Autoevals: EvaluatorInput → Dict[str, Any] (eval kwargs) - Refactor BaseAdapter: remove intermediate execute(), add _run() pattern - 83 unit tests passing

chianhw · 2026-07-08T18:56:48Z

+            customer_mapper=lambda ev: {
+                "input": ev.session_spans[0]["attributes"]["question"],
+                "output": ev.session_spans[0]["attributes"]["answer"],
+                "expected": "the expected answer",
+            },


Do we need to accept aws lambda?

chianhw · 2026-07-08T20:16:27Z

+    def __init__(
+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,


We have not updated this type Dict[str, Any]?

Regarding naming, "Custom mapper" would be a better choice for describing an arbitrary mapper provided by a customer?

chianhw · 2026-07-08T21:00:07Z

+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,
+        threshold: float = 0.5,


label: Optional[str] = None label in EvaluatorOutput is optional.
We don't need to have a default value to generate a label when a metric doesn't have a label and the user doesn't provide any threshold. Can we set default as None?

chianhw · 2026-07-08T21:00:37Z

+                mapping when provided. Expected keys: input, output, expected (optional).
+            threshold: Score threshold for Pass/Fail determination. Defaults to 0.5.
+        """
+        self.scorer = scorer


this naming is not aligned with DeepEval adaptor. Looks like you haven't updated AutoevalsAdapter.
If you haven't completed end to end tests for AutoevalsAdapter, you should not include it in your PR.

chianhw · 2026-07-08T21:00:43Z

+            customer_mapper=lambda ev: LLMTestCase(
+                input=ev.session_spans[0]["attributes"]["user_query"],
+                actual_output=ev.session_spans[0]["attributes"]["response"],
+            ),


using lambda? same as above

chianhw · 2026-07-08T21:00:49Z

+    def __init__(
+        self,
+        metric: BaseMetric,
+        customer_mapper: Optional[Callable[[EvaluatorInput], LLMTestCase]] = None,


naming? same as above

chianhw · 2026-07-08T21:00:52Z

+        self.metric = metric
+        self.customer_mapper = customer_mapper
+        if model is not None:
+            self.metric.model = model


Where do we use self.metric.model?

chianhw · 2026-07-08T21:16:32Z

+    Raises:
+        ValueError: If no mapper can extract data from the spans.
+    """
+    result = _strands_mapper.map(session_spans)


We can check the span scope name and then decide which mapper to use.

haomiao037 added 14 commits June 30, 2026 11:59

Add DeepEvalHandler integration with unit tests

77d16a9

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

Fix span extraction to use real AgentCore _eval_log_records structure

402ea78

Set context field from tool messages for HallucinationMetric support

e9ef47d

Use metric.success for label instead of manual threshold comparison

f97827e

Add model override and timeout enforcement to DeepEvalHandler

9d256ae

Add model override, timeout enforcement, use metric.success, fix Sing…

c142d50

…leTurnParams deprecation

Fix _get_required_params to handle GEval unmappable typing params

3ccc98c

Add .deepeval/ to gitignore

a884f91

Move model override to init to avoid per-call mutation

6ac198c

Refactor to BaseAdapter framework with DeepEval/Autoevals adapters an…

8d415e5

…d EvaluatorInput support

Major refactor: move to custom_code_based_evaluators, add span parser…

8627ab0

… layer, simplify per TJ/Irene feedback

Fix review items: add reference_inputs to model, tighten error detect…

9a9b6a7

…ion, add validate_fields to DeepEvalAdapter

Adapt to upstream ReferenceInput model: remove duplicate field, use e…

0499d4b

…xpected_response_text property

stone-coding requested a review from a team July 6, 2026 17:07

stone-coding requested a deployment to manual-approval July 6, 2026 17:07 — with GitHub Actions Waiting

chianhw reviewed Jul 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Revision third-party eval metrics adapter (DeepEval + Autoevals)#568

feat: Revision third-party eval metrics adapter (DeepEval + Autoevals)#568
stone-coding wants to merge 14 commits into
aws:mainfrom
stone-coding:feature/third-party-eval-adapter

stone-coding commented Jul 6, 2026 •

edited

Loading

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

chianhw Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

stone-coding commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stone-coding commented Jul 6, 2026 •

edited

Loading