feat(eval): grouped help and command examples for the eval commands

[jariy17]

I have decided to implement help groups using the Commander helpGroup feature. As shown below, organizing flags into groups makes it easier for users to understand how each flag functions within the command.

agentcore eval batch-evaluation evaluate

Before (refactor):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              session ground truth (JSON SessionMetadataShape[]; inline, file://<path>, or -)
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Configuration:
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Evaluation:
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              expected answers for the sessions (JSON SessionMetadataShape[])

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "timeRange": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

  --ground-truth (JSON: list of objects)
      Expected answers for the sessions being evaluated, so an evaluator can score a
      response against a reference instead of judging it on its own. Each entry names
      one session; omit an entry for a session that has no reference answer.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "sessionId": "string",              // [required] the session this applies to
            "testScenarioId": "string",         // groups sessions replaying one scenario
            "groundTruth": {                    // exactly one key; only inline today
              "inline": {
                "turns": [                      // the expected exchange, in order
                  {
                    "input": { "prompt": "string" },
                    "expectedResponse": { "text": "string" }
                  },
                  ...
                ],
                "assertions": [                 // statements the response must satisfy
                  { "text": "string" },
                  ...
                ],
                "expectedTrajectory": {
                  "toolNames": ["string", ...]  // tools the agent should have called
                }
              }
            },
            "metadata": { "string": "string", ... }
          },
          ...
        ]

      Example:
        --ground-truth '[{"sessionId":"session-123","groundTruth":{"inline":{"turns":[{"input":{"prompt":"Where is my order?"},"expectedResponse":{"text":"It shipped on Tuesday."}}]}}}]'

        --ground-truth file://ground-truth.json

agentcore eval batch-evaluation simulate

Before (refactor):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Options:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    JSON payload template; {input} is the scenario input, e.g. {"prompt":"{input}"}
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --evaluators <evaluators...>             evaluator ID(s) to apply
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Runtime invocation:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    request body per example (JSON object); {input} is replaced with the input
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID

Dataset:
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)

Configuration:
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest

Evaluation:
  --evaluators <evaluators...>             evaluator ID(s) to apply

Other options:
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

Parameter details:

  --payload-template (JSON object)
      The request body sent to the Runtime for each dataset example. Every occurrence
      of {input} is replaced with that example's input, so the template describes the
      shape your agent expects and {input} marks where the prompt goes.

      Example:
        --payload-template '{"prompt":"{input}"}'

        --payload-template '{"messages":[{"role":"user","content":"{input}"}],"stream":false}'

agentcore eval batch-insights run

Before (refactor):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Configuration:
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Analysis:
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "timeRange": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

agentcore eval online-eval create

Before (refactor):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Options:
  --name <name>                                        the name of the online evaluation config
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)
  --data-source-config <data-source-config>            the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin), as an alternative to --agent
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)
  --description <description>                          a description of the config's monitoring purpose
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Configuration:
  --name <name>                                        the name of the online evaluation config
  --description <description>                          a description of the config's monitoring purpose
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)

Session source (choose exactly one):
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --data-source-config <data-source-config>            the traces to sample (JSON DataSourceConfig), as an alternative to --agent
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

agentcore eval online-eval update

Before (refactor):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Options:
  --id <id>                                            the ID of the online evaluation config to update
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)
  --data-source-config <data-source-config>            replace the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Target:
  --id <id>                                            the ID of the online evaluation config to update

Session source:
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --data-source-config <data-source-config>            replace the traces to sample (JSON DataSourceConfig)

Source filters:
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  replace the trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

Tests

Router (6 new): heading names and ordering, ungrouped flags staying in Options:, -h placement with and without groups, examples rendering exactly once after Parameter details:, backslash continuation for array commands vs single-line for strings, and the no-examples case.

Handlers (14 new, src/handlers/eval/helpGroups.test.ts): asserts what the router's rendering tests cannot — that every flag on these five commands carries a group, that the group comes from the shared vocabulary, that a heading's flags are declared contiguously (a stray one would reorder the headings), and that no example names a flag its command doesn't declare.

Mutation-checked: ungrouping a flag, hand-typing a heading that drifts from the vocabulary, and misspelling a flag inside an example each fail a test.

Verification

• bun test — 3199 pass, 0 fail (225 files)
• bun run typecheck, bun run lint:check, bun run format:check — clean
• No existing test needed changing: the *.screen.test.tsx suites don't assert on these commands' option layout

Follow-ups

#2265 adds --output-config to batch-evaluation, renames --qualifier → --endpoint, and makes --start-time/--end-time require a timezone. #2266 does the online-eval equivalents plus the managed-role widening.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Reviewed the router mechanism (flags.tsx, handler.tsx, router.tsx, index.tsx), the --evaluator → --evaluators rename across the 8 handlers, and the SDK bump.

• The Flag.group plumbing is minimal and correctly optional; ungrouped flags stay in Commander's default section.
• Handler.examples is optional and read off the authored node in compile(), so middleware wrappers (which only need to forward handle) don't have to carry it. withEffectiveTuiSupport also forwards it via handler.examples?.() so nothing gets dropped through TUI wrapping.
• The -h, --help relocation to Other options: correctly triggers only when ownFlags.some(f => f.group), keeping unchanged behavior for commands that don't opt in.
• New router tests cover: heading ordering, ungrouped flags remaining in Options:, -h placement in both cases, examples rendering exactly once after Parameter details:, and the no-examples case.
• The rename is complete: no leftover "--evaluator" argv, flag("evaluator", …), or flags["evaluator"] reads in src/, and the singular evaluator resource-type enum in project remove is correctly left alone (it's a resource selector, not a flag list). Error messages updated to '--evaluators <evaluators...>'. All corresponding fixture/unit tests updated in the same PR.
• SDK bump to 3.1129.0 on both agentcore clients is scoped and justified for the follow-up PRs.

No changes requested.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.06%. Comparing base (551ab96) to head (77bbb9f).
⚠️ Report is 5 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff             @@
##           refactor    #2262    +/-   ##
==========================================
  Coverage     97.05%   97.06%            
==========================================
  Files           566      568     +2     
  Lines         39228    39346   +118     
==========================================
+ Hits          38073    38191   +118     
  Misses         1155     1155            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[notgitika]

added comments for 4 bugs my agent found otherwise LGTM

[notgitika]

this documented shape doesn’t match SessionMetadataShape. groundTruth.inline must be an InlineGroundTruth object, not a string. The current example will produce an invalid API request. Could we update it to use a valid shape such as {"inline":{"turns":[...]}}?

[notgitika]

this property should be timeRange, not sessionFilterConfig. the SDK serializer drops the unknown property, so these dates are silently omitted and the evaluation may include sessions outside the intended window

[jariy17]

The dependenxy will be updated in the next pr

[notgitika]

The locked control-plane SDK does not support logGroupNamePrefixes; CloudWatchLogsInputConfig requires both logGroupNames and serviceNames. Following the documented prefix alternative sends no logGroupNames and fails service validation.
Could we remove that alternative?

[notgitika]

This grouping is lost in the interactive TUI. These eval commands render through CliOnlyScreen, which flattens visibleOptions() into one “options” table without consulting each option’s help group. Could we preserve the groups there as well so the CLI and TUI help remain consistent?

[jariy17]

Sure ill look into this

[jariy17]

Fixed in 77bbb9f6 — CliOnlyScreen now reads each option's helpGroupHeading and renders one section per heading, in the same first-seen order Commander uses, so the CLI and TUI agree. Commands whose flags carry no group keep the single options section unchanged.

 configuration
   --name <name>                batch evaluation name (must be unique in the account)
   --description <description>  optional description
   --kms-key-arn <kms-key-arn>  KMS key to encrypt evaluation data at rest

 session source (choose exactly one)
   --agent <agent>                            harness ID or Runtime ID whose sessions to use
   --online-eval <online-eval>                use sessions an online-eval config already sampled
   --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape
                                              hatch

 source filters
   --endpoint <endpoint>           Runtime endpoint qualifier (default DEFAULT; only with --agent)
   --start-time <start-time>       window start (ISO-8601, with --end-time)
   --end-time <end-time>           window end (ISO-8601, with --start-time)
   --session-ids <session-ids...>  specific session IDs (only with --agent)

 evaluation
   --evaluators <evaluators...>   evaluator ID(s) to apply
   --ground-truth <ground-truth>  expected answers for the sessions (JSON SessionMetadataShape[])

Headings are lowercased and lose the trailing colon so they sit alongside the screen's own arguments heading.

Why not just render Commander's own help text

The obvious simplification is to drop the hand-built table entirely and print command.helpInformation(), which already renders the groups — one source of truth, no duplicated layout. I prototyped it and it does not work here, for a reason worth recording rather than rediscovering:

Help.formatItem skips wrapping when the description column would be too narrow (commander/lib/help.js:663):

const remainingWidth = helpWidth - termWidth - spacerWidth - itemIndent;
if (remainingWidth < this.minWidthToWrap /* 40 */) {
  formattedDescription = description;   // one long unwrapped line
}

termWidth is the width of the command's longest flag term, so the threshold is per-command. At the TUI's usable width of 96:

Command	Term column	Needs helpWidth ≥	At 96
eval batch-evaluation evaluate	41	85	wraps
eval online-eval create	51	95	wraps
harness create	53	97	skipped
gateway create	57	101	skipped

When wrapping is skipped, Commander emits the whole description on one line and Ink re-wraps it with no hanging indent, so continuations land at column 1:

   --protocol <protocol>                                      restrict Target protocols to MCP;
 omitted allows every Target protocol

Raising helpWidth cannot fix gateway create — it needs 101 columns inside a 100-column terminal. Rendering per-option and letting Ink measure each column sidesteps the bailout entirely, and gives tighter columns per section as a side effect.

Worth revisiting if Commander ever exposes minWidthToWrap per call, or if the long-flag commands get shorter terms.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[notgitika]

One remaining blocker: logGroupNamePrefixes is now documented and supported by the upgraded SDK, but the CLI-managed execution role still scopes permissions using only logGroupNames.

For a prefix-only data source, logGroupNamesOf() returns an empty list, so create/update provisions no query access for the matching log groups and will likely fail service validation unless the user supplies --role-arn.

Could we make role scoping prefix-aware and add coverage for prefix-only create/update without a custom role?

Otherwise, my other comments look addressed. TY!

[jariy17]

One remaining blocker: logGroupNamePrefixes is now documented and supported by the upgraded SDK, but the CLI-managed execution role still scopes permissions using only logGroupNames.

For a prefix-only data source, logGroupNamesOf() returns an empty list, so create/update provisions no query access for the matching log groups and will likely fail service validation unless the user supplies --role-arn.

Could we make role scoping prefix-aware and add coverage for prefix-only create/update without a custom role?

I'm adding this support in https://github.com/aws/agentcore-cli/pull/2266. It should be fine for now.