feat(batch-evaluation): add --output-config and rename --qualifier to --endpoint

[jariy17]

Renamed --qualifier to --endpoint in both agentcore eval batch-evaluation simulate and agentcore eval ondemand simulate to match the other commands' runtime-endpoint flag. This is a breaking rename on both commands. We also introduce --output-config to agentcore eval batch-evaluation simulate and agentcore eval batch-evaluation evaluate so customers can control where their results go (docs).

--help

agentcore eval batch-evaluation evaluate

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Configuration:
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Evaluation:
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              expected answers for the sessions (JSON SessionMetadataShape[])

Result output:
  --output-config <output-config>            where results and metrics are written (JSON OutputConfig)

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "timeRange": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

  --ground-truth (JSON: list of objects)
      Expected answers for the sessions being evaluated, so an evaluator can score a
      response against a reference instead of judging it on its own. Each entry names
      one session; omit an entry for a session that has no reference answer.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "sessionId": "string",              // [required] the session this applies to
            "testScenarioId": "string",         // groups sessions replaying one scenario
            "groundTruth": {                    // exactly one key; only inline today
              "inline": {
                "turns": [                      // the expected exchange, in order
                  {
                    "input": { "prompt": "string" },
                    "expectedResponse": { "text": "string" }
                  },
                  ...
                ],
                "assertions": [                 // statements the response must satisfy
                  { "text": "string" },
                  ...
                ],
                "expectedTrajectory": {
                  "toolNames": ["string", ...]  // tools the agent should have called
                }
              }
            },
            "metadata": { "string": "string", ... }
          },
          ...
        ]

      Example:
        --ground-truth '[{"sessionId":"session-123","groundTruth":{"inline":{"turns":[{"input":{"prompt":"Where is my order?"},"expectedResponse":{"text":"It shipped on Tuesday."}}]}}}]'

        --ground-truth file://ground-truth.json

  --output-config (JSON: tagged union object)
      Where evaluation results and metrics are written. Omit it and results go to the
      service-managed default location. Only top-level key: cloudWatchConfig.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchConfig": {
            "logGroupName": "string",      // result log group; omit for
                                           // SOURCE_LOG_GROUP, and it cannot sit
                                           // under /aws/bedrock-agentcore/evaluations/
            "logStreamName": "string",     // result log stream
            "metricsNamespace": "string",  // defaults to
                                           // Bedrock-AgentCore/Evaluations; cannot
                                           // begin with "AWS/"
            "resultDestination": "DEDICATED_LOG_GROUP" | "SOURCE_LOG_GROUP"
                                           // DEDICATED_LOG_GROUP (default) writes to
                                           // a dedicated result group;
                                           // SOURCE_LOG_GROUP writes back to the log
                                           // group the traces were read from
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_OutputConfig.html

      Example:
        --output-config '{"cloudWatchConfig":{"logGroupName":"/company/agent-evaluations","metricsNamespace":"Company/AgentEvaluations","resultDestination":"DEDICATED_LOG_GROUP"}}'

agentcore eval batch-evaluation simulate

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Runtime invocation:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --endpoint <endpoint>                    Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    request body per example (JSON object); {input} is replaced with the input
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID

Dataset:
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)

Configuration:
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest

Evaluation:
  --evaluators <evaluators...>             evaluator ID(s) to apply

Result output:
  --output-config <output-config>          where results and metrics are written (JSON OutputConfig)

Other options:
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

Parameter details:

  --payload-template (JSON object)
      The request body sent to the Runtime for each dataset example. Every occurrence
      of {input} is replaced with that example's input, so the template describes the
      shape your agent expects and {input} marks where the prompt goes.

      Example:
        --payload-template '{"prompt":"{input}"}'

        --payload-template '{"messages":[{"role":"user","content":"{input}"}],"stream":false}'

  --output-config (JSON: tagged union object)
      Where evaluation results and metrics are written. Omit it and results go to the
      service-managed default location. Only top-level key: cloudWatchConfig.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchConfig": {
            "logGroupName": "string",      // result log group; omit for
                                           // SOURCE_LOG_GROUP, and it cannot sit
                                           // under /aws/bedrock-agentcore/evaluations/
            "logStreamName": "string",     // result log stream
            "metricsNamespace": "string",  // defaults to
                                           // Bedrock-AgentCore/Evaluations; cannot
                                           // begin with "AWS/"
            "resultDestination": "DEDICATED_LOG_GROUP" | "SOURCE_LOG_GROUP"
                                           // DEDICATED_LOG_GROUP (default) writes to
                                           // a dedicated result group;
                                           // SOURCE_LOG_GROUP writes back to the log
                                           // group the traces were read from
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_OutputConfig.html

      Example:
        --output-config '{"cloudWatchConfig":{"logGroupName":"/company/agent-evaluations","metricsNamespace":"Company/AgentEvaluations","resultDestination":"DEDICATED_LOG_GROUP"}}'

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Nice, well-scoped PR. The three changes are cleanly separated, the tests cover each layer where it can fail (handler, SessionSource timestamp parsing, and — deliberately — the Core→SDK seam that the TestCoreClient and fixture suites can't cover), and the "resolve --output-config before any Runtime invocation" ordering in simulate is exactly the right call for a command that bills per example. The BatchOutputConfig module and long-form parameter help follow existing conventions and are appropriately API-shaped.

No blocking issues. A couple of small notes, take or leave:

• eval ondemand simulate still uses --qualifier (src/handlers/eval/ondemand/simulate/index.tsx:19,69,92). The PR body justifies the rename by aligning simulate with its own command family (eval), but this sibling command in the same family is left inconsistent. If it's intentionally deferred, a follow-up TODO/issue reference would help; otherwise consider renaming it in the same breaking change so users only see one flag flip.
• Timestamp regex accepts HH:MM with no seconds (sessionSource.tsx:171) — that's fine and matches ISO-8601, but the error message example (2026-09-01T00:00:00Z) and all tests use the seconds form; worth confirming 2026-09-01T00:00Z is intentionally allowed (looks like it is, and it's a superset of what customers will type).

Live verification, mutation-testing notes, and the explanation for the new src/core/eval.test.ts file (the gap between handler-level TestCoreClient assertions and re-record-only fixture assertions) are all appreciated.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.07%. Comparing base (df85f30) to head (8ab7dd6).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #2265   +/-   ##
=========================================
  Coverage     97.07%   97.07%           
=========================================
  Files           571      572    +1     
  Lines         39442    39478   +36     
=========================================
+ Hits          38287    38323   +36     
  Misses         1155     1155           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Use Golden tests.

[jariy17]

I'll fix this TODO later in a follow up PR.

[jariy17]

We don't need this

[jariy17]

remove this code comment

[notgitika]

Looks solid! my agent found some bugs that I added in the comments.

apart from this, the PR title still mentions requiring timestamp timezones, but the current diff and commit msg say that change was dropped. lets update the PR title

[notgitika]

SourceResolver owns the one-stdin-option guard, but this creates a new instance separate from the resolvers used by SessionSource and --ground-truth in evaluate. With --ground-truth - --output-config - (or --data-source-config - --output-config -), the first option drains stdin and this one receives an empty string instead of reporting the conflict. Could we share one resolver or prevalidate that only one option uses stdin, and add a regression test?

[jariy17]

We will share one resolved then and I'll add a regression test!

[notgitika]

This downgrades @aws-sdk/client-bedrock-agentcore from 3.1131.0 to 3.1130.0, and the control client is downgraded on the next entry as well, while package.json is unchanged. Could we regenerate the lockfile or otherwise preserve the parent branch's 3.1131.0 resolutions to avoid an unrelated dependency regression?

[jariy17]

fixed

[notgitika]

This also makes the breaking --qualifier → --endpoint rename for on-demand simulate, while the PR description only calls out batch-evaluation simulate. Please document both affected commands. Should we retain --qualifier as a deprecated alias to avoid breaking existing scripts?

[jariy17]

I'll update the PR description and we haven't released the CLI to the public so it's okay to make breaking changes :)

[notgitika]

Yeah makes sense to me I think we wanted to use --endpoint anyway AFAIK but I wanted to just point it out

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[notgitika]

Thanks for addressing the comments LGTM!

[nborges-aws]

PR LGTM after fixes. Thanks!