feat(online-eval): add --output-config and --tags, --description on update

[jariy17]

Fourth and last of the stack. Now based directly on refactor (#2265 and #2294 have merged).

--output-config on create and update

Same passthrough contract as batch evaluation's flag of the same name: inline JSON, file://<path>, or -, parsed and forwarded with field names and values untouched, rejected before any side effect if malformed.

A separate module and a separate generated type, deliberately. The online-evaluation OutputConfig is a plain object with no logStreamName; the batch one is a tagged union that has one. An online evaluation runs continuously rather than as one job with one stream, so sharing the type would be wrong even though the flag name matches.

--tags on create, --description on update

--tags reuses TagsSchema + parseJsonFlagWithSchema, matching project add memory, so {"team":42} is rejected by the CLI rather than by the API. --description on update was simply missing — create had it.

The execution role, which is what makes the destination usable

At create, executionPolicy granted result writes only to /aws/bedrock-agentcore/evaluations/*. A CLI-managed role plus a custom log group would therefore have created a config whose results could not be written anywhere — the flag would have looked like it worked. So the create-time policy now also grants:

• the customer-named group, for DEDICATED_LOG_GROUP
• the sampled source groups, for SOURCE_LOG_GROUP

update never provisions or re-scopes the execution role — it forwards a caller's --role-arn as-is and leaves the policy to them, matching harness update and the pattern #2294 established for online-eval. So the destination-aware widening applies at create only.

Two invariants worth reviewing closely

A single ARN stays a bare string, not a one-element array. IAM treats those identically, but this document's exact text is hashed to name the inline policy (scopePolicyName). Wrapping it would rename the policy attached to every existing config with no custom destination, orphaning the grant it is currently running on. My first attempt did exactly that, and the fixture suite caught it — onlineEvalExecutionRole.test.ts now pins it.

A destination already inside the reserved /aws/bedrock-agentcore/evaluations/ namespace contributes nothing, since the wildcard covers it. That keeps the document stable for a config whose stored outputConfig is the service-managed default the API echoes back — otherwise the same orphaning problem would hit every config ever created through the CLI.

One shared SourceResolver per command

create and update each build a single SourceResolver and thread it through every stdin-capable flag (--tags, --filters, --data-source-config, --output-config). The resolver's one-stdin guard then fires across options: --filters - --output-config - is reported as a conflict instead of the second flag silently reading an empty string after the first drains stdin.

Live verification — an exploratory account, us-west-2

Create with a custom dedicated group and tags → the config came back with logGroupName: /company/pr4-online-eval-results, and the CLI-provisioned role's policy granted exactly the intended scope:

{
  "Sid": "WriteEvaluationResults",
  "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents"],
  "Resource": [
    "arn:aws:logs:us-west-2:123456789012:log-group:/aws/bedrock-agentcore/evaluations/*",
    "arn:aws:logs:us-west-2:123456789012:log-group:/company/pr4-online-eval-results*"
  ]
}

Tags landed: {"purpose": "pr4-verify", "team": "agentcore-cli"}.

Update with a new --description and --output-config → both reached the service and a follow-up get confirmed them; the execution role was left untouched, as intended.

Cleanup: the config, the provisioned IAM role and its inline policy, and the /company/pr4-online-eval-results log group were all deleted (verified absent).

Tests

• online-eval.flags.test.tsx (new, 13 tests) — output config from inline / stdin reaching Core unchanged on both commands; tags parsed to a map; a non-string tag value rejected; omitted flags staying undefined; malformed --output-config or --tags failing with an empty Core call log, which is the assertion that no IAM role was provisioned for a config that was never created; --description reaching Core; and, on each command, a second option reading from stdin rejected before any SDK call.
• onlineEvalExecutionRole.test.ts — the bare-string invariant, the custom dedicated group, SOURCE_LOG_GROUP scoping to the source prefix, a reserved-namespace destination adding nothing, and a destination change producing a different policy name.

Verification

• bun test — 3242 pass, 0 fail (228 files)
• bun run typecheck, bun run lint:check, bun run format:check — clean
• Existing online-eval fixture suite passes unchanged, which is the backward-compatibility evidence for the policy document

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Changes requested

Nice, careful change — the invariants around the hashed policy name (bare-string vs. one-element array, reserved-namespace destinations, output-included policy fingerprint) are well thought out and well tested. One user-facing correctness issue in the update path warrants a change before merge.

stale-scope warning is wrong when only the output destination moved

src/handlers/eval/online-eval/update/index.tsx (lines ~200–207) still hard-codes the old wording:

if (reason === "stale-scope") {
  io.stderr.write(
    `warning: the execution role still grants access to the previous data source.\n` +
      `  role: ${roleArn}\n` +
      `  detach the inline policy covering: ${logGroupNames.join(", ")}\n`,
  );
}

Now that a --output-config change alone can reach the refresh path in src/core/eval.tsx (via outputMoved → refreshManagedRole), stale-scope is reachable for an output-only move. In that case:

• oldLogGroups === newLogGroups (data source unchanged), so logGroupNames: oldLogGroups in the warning payload lists the current data-source groups — which are not stale at all.
• The message tells the caller the role "still grants access to the previous data source," but the actually-stale portion of the old inline policy is the previous output destination.

So a customer who moves only their result destination will see a warning that names log groups that are still in use and blames the wrong half.

A few ways to fix:

1. Have the handler branch on scope for stale-scope too, mirroring the non-stale wording (e.g. `the ${MOVED[scope]} moved but the superseded inline policy could not be detached`), and populate the warning's logGroupNames in eval.tsx with the groups that were actually stale — for output-only moves that's destinationLogGroupNames(current.outputConfig, current.dataSourceConfig); for input-only it's oldLogGroups; for input-and-output it's the union.
2. Or, at minimum, keep the payload but broaden the message to not claim "data source" — something like "the execution role still grants a superseded scope" and let the caller detach the old inline policy by hash.

Option 1 is the more consistent fix given how the custom-role / update-declined branches already carry scope and format accordingly.

Non-blocking observation (won't block merge)

outputMoved = update.outputConfig !== undefined treats any --output-config as a move, even when it matches the stored config. That's fine for the managed-role refresh (idempotent by hash), but for the custom-role / update-declined branches it can emit a warning telling the customer the "output destination moved" when it didn't. The comment acknowledges the trade-off; not a change request, just worth knowing that the two warning branches also inherit it.

Everything else — the bare-string invariant, reserved-namespace suppression, effectiveOutputConfig for source-only moves, the pre-Core resolution of --output-config and --tags so a malformed JSON can't leave an IAM role behind, and the TestCoreClient.setOnlineEvalRoleScopeWarning seam — looks good.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.99%. Comparing base (25a7a82) to head (b324c4e).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #2266   +/-   ##
=========================================
  Coverage     96.98%   96.99%           
=========================================
  Files           579      580    +1     
  Lines         39442    39515   +73     
=========================================
+ Hits          38253    38326   +73     
  Misses         1189     1189           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Do we need to add permissions to write back in a service log?

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)