Building an evaluation sample on top of bedrock-agentcore 1.23.0, I ended up reimplementing three things the SDK already has internally, because they are not reachable from a public import. Each one is small on its own; together they are the difference between a sample importing the SDK and a sample duplicating it.
1. WaitConfig has no public import path.
AgentCoreRuntimeClient.create_agent_runtime_and_wait(wait_config=...) is public and documented, but its argument type only exists at bedrock_agentcore._utils.polling.WaitConfig. Neither bedrock_agentcore nor bedrock_agentcore.runtime exports it:
from bedrock_agentcore.runtime import WaitConfig
# ImportError: cannot import name 'WaitConfig' from 'bedrock_agentcore.runtime'
So a caller who needs a timeout other than the default 300 seconds has to import from _utils. I chose to accept the default instead, which means a slow create can now time out where my own loop would have waited longer.
Suggestion: export WaitConfig from bedrock_agentcore.runtime alongside AgentCoreRuntimeClient.
2. Tool-span detection is private, and every span consumer needs it.
EvaluationClient._is_tool_span recognises the three conventions in use:
attrs.get("gen_ai.operation.name") == "execute_tool"
or attrs.get("openinference.span.kind") == "TOOL"
or attrs.get("traceloop.span.kind") == "tool"
Anyone reading spans for their own metrics needs exactly this and cannot call it. I wrote a character-for-character copy, which now silently drifts if the SDK adds a fourth convention. _extract_tool_span_ids and _get_evaluator_level have the same problem.
Suggestion: a public bedrock_agentcore.evaluation.spans module with is_tool_span(span), tool_span_ids(spans, trace_id=None) and evaluator_level(evaluator_id).
3. Insights are not reachable through BatchEvaluationRunner.
StartBatchEvaluation accepts an insights list (Builtin.Insight.FailureAnalysis, UserIntent, ExecutionSummary), and those insights are what make the improvement loop usable: they cluster failures and tag root causes as prompt-addressable. BatchEvaluationRunner covers the evaluator path but grepping the evaluation package for "insight" returns nothing, so the insights path stays raw boto3 plus a hand-written poll loop.
Suggestion: support insights in BatchEvaluationRunner, or add a runner for them, including the constraint that insights and evaluators cannot be requested in the same job.
Minor: the client constructors disagree on the region argument.
ConfigBundleClient(region_name="us-west-2") # region_name
AgentCoreRuntimeClient(region="us-west-2") # region
AgentCoreRuntimeClient(region_name=...) raises TypeError. Worth aligning, or accepting both.
Versions: bedrock-agentcore 1.23.0, Python 3.13, us-west-2.
Building an evaluation sample on top of
bedrock-agentcore1.23.0, I ended up reimplementing three things the SDK already has internally, because they are not reachable from a public import. Each one is small on its own; together they are the difference between a sample importing the SDK and a sample duplicating it.1.
WaitConfighas no public import path.AgentCoreRuntimeClient.create_agent_runtime_and_wait(wait_config=...)is public and documented, but its argument type only exists atbedrock_agentcore._utils.polling.WaitConfig. Neitherbedrock_agentcorenorbedrock_agentcore.runtimeexports it:So a caller who needs a timeout other than the default 300 seconds has to import from
_utils. I chose to accept the default instead, which means a slow create can now time out where my own loop would have waited longer.Suggestion: export
WaitConfigfrombedrock_agentcore.runtimealongsideAgentCoreRuntimeClient.2. Tool-span detection is private, and every span consumer needs it.
EvaluationClient._is_tool_spanrecognises the three conventions in use:Anyone reading spans for their own metrics needs exactly this and cannot call it. I wrote a character-for-character copy, which now silently drifts if the SDK adds a fourth convention.
_extract_tool_span_idsand_get_evaluator_levelhave the same problem.Suggestion: a public
bedrock_agentcore.evaluation.spansmodule withis_tool_span(span),tool_span_ids(spans, trace_id=None)andevaluator_level(evaluator_id).3. Insights are not reachable through
BatchEvaluationRunner.StartBatchEvaluationaccepts aninsightslist (Builtin.Insight.FailureAnalysis,UserIntent,ExecutionSummary), and those insights are what make the improvement loop usable: they cluster failures and tag root causes as prompt-addressable.BatchEvaluationRunnercovers the evaluator path but grepping theevaluationpackage for "insight" returns nothing, so the insights path stays raw boto3 plus a hand-written poll loop.Suggestion: support insights in
BatchEvaluationRunner, or add a runner for them, including the constraint that insights and evaluators cannot be requested in the same job.Minor: the client constructors disagree on the region argument.
AgentCoreRuntimeClient(region_name=...)raisesTypeError. Worth aligning, or accepting both.Versions:
bedrock-agentcore1.23.0, Python 3.13, us-west-2.