feat(project): add --model-provider to project add evaluator llm-as-a-judge

[jariy17]

What

Adds --model-provider (Bedrock default / OpenResponses) to agentcore project add evaluator llm-as-a-judge, so a project can declare an OpenResponses judge alongside the existing Bedrock one.

Part 1 of 2. PR 2 (stacked) adds the same to the imperative eval evaluator llm-as-a-judge create/update commands and makes updates provider-safe.

Changes

• Schema (src/projectSchemas/evaluator.ts): EvaluatorModelProviderSchema (Bedrock | OpenResponses), a provider-neutral EvaluatorModelIdSchema, isValidOpenResponsesModelId, and an optional modelProvider on LlmAsAJudgeConfigSchema.
• Handler (project add evaluator llm-as-a-judge): --model-provider flag, defaults to Bedrock, validates the model per resolved provider, and persists modelProvider only for OpenResponses.

Compatibility

• Bedrock stays the default and remains implicit in agentcore.json — existing Bedrock projects require no changes.
• OpenResponses model IDs are validated as non-empty printable identifiers without spaces (no hard-coded catalog).

Verification

• bun test src/projectSchemas/evaluator.test.ts + .../add/evaluator/llm-as-a-judge/index.test.ts — 30 pass.
• bun run typecheck + bun run lint:check clean.
• --help lists Bedrock (default) and OpenResponses.

Bug-bash recording against the explore account to follow in a comment.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Changes requested

Main issue: OpenResponses cannot actually be deployed

This PR teaches the CLI to accept --model-provider OpenResponses and persist modelProvider: "OpenResponses" into agentcore.json, but the deployment path in aws/agentcore-l3-cdk-constructs doesn't know about this field:

• src/schema/schemas/primitives/evaluator.ts — LlmAsAJudgeConfigSchema has no modelProvider field. Default zod objects strip unknown keys, so on deploy the field is silently dropped.
• src/cdk/constructs/components/primitives/evaluator/AgentCoreEvaluator.ts (around line 92–108) unconditionally emits evaluatorConfig.llmAsAJudge.modelConfig.bedrockEvaluatorModelConfig.modelId = llmAsAJudge.model. There is no OpenResponses branch.

The user-visible consequence is that agentcore add evaluator llm-as-a-judge --model-provider OpenResponses --model openai.gpt-5.4 ... will happily write the project file, but agentcore deploy will pass openai.gpt-5.4 as a Bedrock model ID and CloudFormation will fail (or, worse, silently try to invoke it against Bedrock). There's nothing between the CLI validator and CFN that tells the user the deploy path doesn't support OpenResponses.

Options to fix:

1. Land the L3 side first (preferred). Add modelProvider to LlmAsAJudgeConfigSchema in agentcore-l3-cdk-constructs, teach AgentCoreEvaluator to select the appropriate modelConfig variant (openResponsesEvaluatorModelConfig/whatever the CFN resource exposes), cut a release, bump the pinned version in the CLI, then merge this PR.
2. Gate the flag behind the L3 gaining support. Keep the schema change, but have the handler throw InputValidationError for --model-provider OpenResponses with a "not yet supported end-to-end" message until the L3 catches up. Avoids shipping a footgun.
3. Reject at deploy time. Add a validation step in the deploy flow (e.g. in handlers/project/deploy or the CDK backend's spec check) that fails fast when it encounters llmAsAJudge.modelProvider === "OpenResponses" while the pinned L3 version doesn't support it. Less nice than option 1 or 2 because failure happens later.

Please pick one — as it stands the new flag is functionally advertising a capability the product doesn't have.

Nit-level (not blocking)

Not flagging any of the style items. Tests and validation logic themselves look good and use real temp projects (no fs mocking), matching the harness guidance.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.03%. Comparing base (53f13f6) to head (551e07b).

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #2298   +/-   ##
=========================================
  Coverage     97.03%   97.03%           
=========================================
  Files           580      580           
  Lines         39630    39659   +29     
=========================================
+ Hits          38453    38482   +29     
  Misses         1177     1177           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Bug bash — project add evaluator (live)

Recorded with the TUI harness in a scratch project.

• project add evaluator llm-as-a-judge without --model-provider → Bedrock, and modelProvider is absent from agentcore.json (existing Bedrock projects are byte-for-byte unchanged).
• --model-provider OpenResponses --model openai.gpt-5.4 → modelProvider: "OpenResponses" persisted explicitly.
• Verified via a jq view of the written spec: the Bedrock evaluator shows modelProvider: null (key absent), the OpenResponses one shows modelProvider: "OpenResponses".

Validation (--model-provider value, per-provider --model format) is covered by unit tests; invalid combinations are rejected before the spec is touched.

A screen recording of this run is archived internally (Artifactory); it is not linked here because this repository is public.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)