Skip to content

Span helpers, WaitConfig and batch insights are not reachable from a public import #663

Description

@sundargthb

Building an evaluation sample on top of bedrock-agentcore 1.23.0, I ended up reimplementing three things the SDK already has internally, because they are not reachable from a public import. Each one is small on its own; together they are the difference between a sample importing the SDK and a sample duplicating it.

1. WaitConfig has no public import path.

AgentCoreRuntimeClient.create_agent_runtime_and_wait(wait_config=...) is public and documented, but its argument type only exists at bedrock_agentcore._utils.polling.WaitConfig. Neither bedrock_agentcore nor bedrock_agentcore.runtime exports it:

from bedrock_agentcore.runtime import WaitConfig
# ImportError: cannot import name 'WaitConfig' from 'bedrock_agentcore.runtime'

So a caller who needs a timeout other than the default 300 seconds has to import from _utils. I chose to accept the default instead, which means a slow create can now time out where my own loop would have waited longer.

Suggestion: export WaitConfig from bedrock_agentcore.runtime alongside AgentCoreRuntimeClient.

2. Tool-span detection is private, and every span consumer needs it.

EvaluationClient._is_tool_span recognises the three conventions in use:

attrs.get("gen_ai.operation.name") == "execute_tool"
or attrs.get("openinference.span.kind") == "TOOL"
or attrs.get("traceloop.span.kind") == "tool"

Anyone reading spans for their own metrics needs exactly this and cannot call it. I wrote a character-for-character copy, which now silently drifts if the SDK adds a fourth convention. _extract_tool_span_ids and _get_evaluator_level have the same problem.

Suggestion: a public bedrock_agentcore.evaluation.spans module with is_tool_span(span), tool_span_ids(spans, trace_id=None) and evaluator_level(evaluator_id).

3. Insights are not reachable through BatchEvaluationRunner.

StartBatchEvaluation accepts an insights list (Builtin.Insight.FailureAnalysis, UserIntent, ExecutionSummary), and those insights are what make the improvement loop usable: they cluster failures and tag root causes as prompt-addressable. BatchEvaluationRunner covers the evaluator path but grepping the evaluation package for "insight" returns nothing, so the insights path stays raw boto3 plus a hand-written poll loop.

Suggestion: support insights in BatchEvaluationRunner, or add a runner for them, including the constraint that insights and evaluators cannot be requested in the same job.

Minor: the client constructors disagree on the region argument.

ConfigBundleClient(region_name="us-west-2")     # region_name
AgentCoreRuntimeClient(region="us-west-2")      # region

AgentCoreRuntimeClient(region_name=...) raises TypeError. Worth aligning, or accepting both.

Versions: bedrock-agentcore 1.23.0, Python 3.13, us-west-2.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions